[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TimestampAdvancementITest
[ RUN ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260317 21:02:58.961651 29677 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.251.126:34775
I20260317 21:02:58.961998 29677 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:02:58.962193 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:02:58.966439 29685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:02:58.966471 29683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:02:58.966708 29682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:02:58.966922 29677 server_base.cc:1060] running on GCE node
I20260317 21:02:58.967183 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:02:58.967227 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:02:58.967249 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781378967250 us; error 0 us; skew 500 ppm
I20260317 21:02:58.968412 29677 webserver.cc:492] Webserver started at http://127.28.251.126:38671/ using document root <none> and password file <none>
I20260317 21:02:58.968681 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:02:58.968782 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:02:58.968904 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:02:58.970039 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/master-0-root/instance:
uuid: "5401cbf7a89e418baf84f08087c1c527"
format_stamp: "Formatted at 2026-03-17 21:02:58 on dist-test-slave-tm0n"
I20260317 21:02:58.971537 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260317 21:02:58.972208 29690 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:58.972450 29677 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260317 21:02:58.972544 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/master-0-root
uuid: "5401cbf7a89e418baf84f08087c1c527"
format_stamp: "Formatted at 2026-03-17 21:02:58 on dist-test-slave-tm0n"
I20260317 21:02:58.972638 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:02:58.990716 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:02:58.991045 29677 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:02:58.991190 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:02:58.991560 29696 maintenance_manager.cc:346] Maintenance manager is disabled (check --enable_maintenance_manager).
I20260317 21:02:58.995702 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.126:34775
I20260317 21:02:58.995787 29742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.126:34775 every 8 connection(s)
I20260317 21:02:58.996361 29743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:02:58.998876 29743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527: Bootstrap starting.
I20260317 21:02:58.999740 29743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527: Neither blocks nor log segments found. Creating new log.
I20260317 21:02:59.000005 29743 log.cc:826] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527: Log is configured to *not* fsync() on all Append() calls
I20260317 21:02:59.000649 29743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527: No bootstrap required, opened a new log
I20260317 21:02:59.001961 29743 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401cbf7a89e418baf84f08087c1c527" member_type: VOTER }
I20260317 21:02:59.002065 29743 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:02:59.002085 29743 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5401cbf7a89e418baf84f08087c1c527, State: Initialized, Role: FOLLOWER
I20260317 21:02:59.002224 29743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401cbf7a89e418baf84f08087c1c527" member_type: VOTER }
I20260317 21:02:59.002301 29743 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 21:02:59.002323 29743 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 21:02:59.002388 29743 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:02:59.002933 29743 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401cbf7a89e418baf84f08087c1c527" member_type: VOTER }
I20260317 21:02:59.003031 29743 leader_election.cc:304] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5401cbf7a89e418baf84f08087c1c527; no voters:
I20260317 21:02:59.003208 29743 leader_election.cc:290] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260317 21:02:59.003360 29746 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:02:59.003465 29743 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:02:59.003536 29746 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 1 LEADER]: Becoming Leader. State: Replica: 5401cbf7a89e418baf84f08087c1c527, State: Running, Role: LEADER
I20260317 21:02:59.003686 29746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401cbf7a89e418baf84f08087c1c527" member_type: VOTER }
I20260317 21:02:59.004061 29748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5401cbf7a89e418baf84f08087c1c527. Latest consensus state: current_term: 1 leader_uuid: "5401cbf7a89e418baf84f08087c1c527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401cbf7a89e418baf84f08087c1c527" member_type: VOTER } }
I20260317 21:02:59.004098 29747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5401cbf7a89e418baf84f08087c1c527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5401cbf7a89e418baf84f08087c1c527" member_type: VOTER } }
I20260317 21:02:59.004230 29747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [sys.catalog]: This master's current role is: LEADER
I20260317 21:02:59.004479 29748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [sys.catalog]: This master's current role is: LEADER
I20260317 21:02:59.004513 29754 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:02:59.004935 29754 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:02:59.005121 29677 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260317 21:02:59.006631 29754 catalog_manager.cc:1357] Generated new cluster ID: 66491192548248cbb4f62e3d21cda14b
I20260317 21:02:59.006695 29754 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:02:59.015132 29754 catalog_manager.cc:1380] Generated new certificate authority record
I20260317 21:02:59.015705 29754 catalog_manager.cc:1514] Loading token signing keys...
I20260317 21:02:59.024051 29754 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527: Generated new TSK 0
I20260317 21:02:59.024293 29754 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:02:59.037395 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260317 21:02:59.037602 29677 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260317 21:02:59.039307 29765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:02:59.039443 29766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:02:59.039546 29768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:02:59.039710 29677 server_base.cc:1060] running on GCE node
I20260317 21:02:59.039817 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:02:59.039852 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:02:59.039872 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781379039872 us; error 0 us; skew 500 ppm
I20260317 21:02:59.040537 29677 webserver.cc:492] Webserver started at http://127.28.251.65:40655/ using document root <none> and password file <none>
I20260317 21:02:59.040644 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:02:59.040695 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:02:59.040738 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:02:59.041060 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-0-root/instance:
uuid: "9f1d952e629c4010ae48963a9cdee04a"
format_stamp: "Formatted at 2026-03-17 21:02:59 on dist-test-slave-tm0n"
I20260317 21:02:59.042303 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260317 21:02:59.042892 29773 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.043059 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260317 21:02:59.043115 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-0-root
uuid: "9f1d952e629c4010ae48963a9cdee04a"
format_stamp: "Formatted at 2026-03-17 21:02:59 on dist-test-slave-tm0n"
I20260317 21:02:59.043162 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:02:59.052270 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:02:59.052623 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:02:59.052846 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:02:59.053244 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:02:59.053292 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.053325 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:02:59.053344 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.058023 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.65:41093
I20260317 21:02:59.058108 29836 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.65:41093 every 8 connection(s)
I20260317 21:02:59.058429 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260317 21:02:59.058590 29677 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260317 21:02:59.062333 29840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:02:59.062842 29843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:02:59.062944 29677 server_base.cc:1060] running on GCE node
W20260317 21:02:59.063047 29841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:02:59.063227 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:02:59.063287 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:02:59.063309 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781379063309 us; error 0 us; skew 500 ppm
I20260317 21:02:59.063918 29677 webserver.cc:492] Webserver started at http://127.28.251.66:45919/ using document root <none> and password file <none>
I20260317 21:02:59.064031 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:02:59.064095 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:02:59.064167 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:02:59.064473 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-1-root/instance:
uuid: "24015e78ad6341a9a2bc13754a55b581"
format_stamp: "Formatted at 2026-03-17 21:02:59 on dist-test-slave-tm0n"
I20260317 21:02:59.065486 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260317 21:02:59.067039 29837 heartbeater.cc:344] Connected to a master server at 127.28.251.126:34775
I20260317 21:02:59.067138 29837 heartbeater.cc:461] Registering TS with master...
I20260317 21:02:59.067353 29837 heartbeater.cc:507] Master 127.28.251.126:34775 requested a full tablet report, sending...
I20260317 21:02:59.067843 29707 ts_manager.cc:194] Registered new tserver with Master: 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093)
I20260317 21:02:59.068516 29707 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48226
I20260317 21:02:59.069284 29849 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.069504 29677 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260317 21:02:59.069569 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-1-root
uuid: "24015e78ad6341a9a2bc13754a55b581"
format_stamp: "Formatted at 2026-03-17 21:02:59 on dist-test-slave-tm0n"
I20260317 21:02:59.069648 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:02:59.085042 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:02:59.085382 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:02:59.085589 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:02:59.085804 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:02:59.085853 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.085897 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:02:59.085924 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.089954 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.66:33653
I20260317 21:02:59.090021 29912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.66:33653 every 8 connection(s)
I20260317 21:02:59.090375 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260317 21:02:59.090529 29677 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260317 21:02:59.091787 29917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:02:59.091888 29916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:02:59.094291 29913 heartbeater.cc:344] Connected to a master server at 127.28.251.126:34775
I20260317 21:02:59.094376 29913 heartbeater.cc:461] Registering TS with master...
I20260317 21:02:59.094497 29913 heartbeater.cc:507] Master 127.28.251.126:34775 requested a full tablet report, sending...
I20260317 21:02:59.094741 29707 ts_manager.cc:194] Registered new tserver with Master: 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:02:59.095091 29707 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48230
W20260317 21:02:59.098304 29919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:02:59.098394 29677 server_base.cc:1060] running on GCE node
I20260317 21:02:59.098598 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:02:59.098662 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:02:59.098694 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781379098694 us; error 0 us; skew 500 ppm
I20260317 21:02:59.099328 29677 webserver.cc:492] Webserver started at http://127.28.251.67:43303/ using document root <none> and password file <none>
I20260317 21:02:59.099458 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:02:59.099531 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:02:59.099603 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:02:59.099896 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-2-root/instance:
uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
format_stamp: "Formatted at 2026-03-17 21:02:59 on dist-test-slave-tm0n"
I20260317 21:02:59.100818 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260317 21:02:59.101372 29924 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.101548 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.101608 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-2-root
uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
format_stamp: "Formatted at 2026-03-17 21:02:59 on dist-test-slave-tm0n"
I20260317 21:02:59.101660 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:02:59.124280 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:02:59.124609 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:02:59.124795 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:02:59.125038 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:02:59.125088 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.125133 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:02:59.125162 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.129047 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.67:33237
I20260317 21:02:59.129113 29987 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.67:33237 every 8 connection(s)
I20260317 21:02:59.132869 29988 heartbeater.cc:344] Connected to a master server at 127.28.251.126:34775
I20260317 21:02:59.132962 29988 heartbeater.cc:461] Registering TS with master...
I20260317 21:02:59.133088 29988 heartbeater.cc:507] Master 127.28.251.126:34775 requested a full tablet report, sending...
I20260317 21:02:59.133394 29707 ts_manager.cc:194] Registered new tserver with Master: 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:02:59.133839 29677 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004580437s
I20260317 21:02:59.133906 29707 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48234
I20260317 21:02:59.152952 29707 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48260:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:02:59.153262 29707 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:02:59.159677 29877 tablet_service.cc:1511] Processing CreateTablet for tablet e2b6dd1d6b8c42a9999eed01df93529f (DEFAULT_TABLE table=test-workload [id=9ab734d60012407680be58d55fc874b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:02:59.159885 29877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2b6dd1d6b8c42a9999eed01df93529f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:02:59.159965 29952 tablet_service.cc:1511] Processing CreateTablet for tablet e2b6dd1d6b8c42a9999eed01df93529f (DEFAULT_TABLE table=test-workload [id=9ab734d60012407680be58d55fc874b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:02:59.160084 29952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2b6dd1d6b8c42a9999eed01df93529f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:02:59.161113 30009 tablet_bootstrap.cc:492] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Bootstrap starting.
I20260317 21:02:59.161141 29801 tablet_service.cc:1511] Processing CreateTablet for tablet e2b6dd1d6b8c42a9999eed01df93529f (DEFAULT_TABLE table=test-workload [id=9ab734d60012407680be58d55fc874b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:02:59.161605 29801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2b6dd1d6b8c42a9999eed01df93529f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:02:59.161648 30009 tablet_bootstrap.cc:654] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Neither blocks nor log segments found. Creating new log.
I20260317 21:02:59.162319 30009 tablet_bootstrap.cc:492] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: No bootstrap required, opened a new log
I20260317 21:02:59.162364 30009 ts_tablet_manager.cc:1403] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260317 21:02:59.162528 30009 raft_consensus.cc:359] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.162598 30009 raft_consensus.cc:385] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:02:59.162619 30009 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Initialized, Role: FOLLOWER
I20260317 21:02:59.162674 30009 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.162858 30009 ts_tablet_manager.cc:1434] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260317 21:02:59.162935 29913 heartbeater.cc:499] Master 127.28.251.126:34775 was elected leader, sending a full tablet report...
I20260317 21:02:59.163067 30013 tablet_bootstrap.cc:492] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Bootstrap starting.
I20260317 21:02:59.163980 30013 tablet_bootstrap.cc:654] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Neither blocks nor log segments found. Creating new log.
I20260317 21:02:59.164378 30010 tablet_bootstrap.cc:492] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Bootstrap starting.
I20260317 21:02:59.164582 30013 tablet_bootstrap.cc:492] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: No bootstrap required, opened a new log
I20260317 21:02:59.164638 30013 ts_tablet_manager.cc:1403] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260317 21:02:59.164839 30010 tablet_bootstrap.cc:654] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Neither blocks nor log segments found. Creating new log.
I20260317 21:02:59.164863 30013 raft_consensus.cc:359] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.165010 30013 raft_consensus.cc:385] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:02:59.165067 30013 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Initialized, Role: FOLLOWER
I20260317 21:02:59.165151 30013 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.165385 30013 ts_tablet_manager.cc:1434] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260317 21:02:59.165457 29837 heartbeater.cc:499] Master 127.28.251.126:34775 was elected leader, sending a full tablet report...
I20260317 21:02:59.165416 30010 tablet_bootstrap.cc:492] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: No bootstrap required, opened a new log
I20260317 21:02:59.165566 30010 ts_tablet_manager.cc:1403] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260317 21:02:59.165711 30010 raft_consensus.cc:359] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.165774 30010 raft_consensus.cc:385] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:02:59.165800 30010 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Initialized, Role: FOLLOWER
I20260317 21:02:59.165859 30010 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.166009 30010 ts_tablet_manager.cc:1434] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260317 21:02:59.166110 29988 heartbeater.cc:499] Master 127.28.251.126:34775 was elected leader, sending a full tablet report...
I20260317 21:02:59.179997 30012 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:02:59.180106 30012 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.180346 30012 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:02:59.183365 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" is_pre_election: true
I20260317 21:02:59.183532 29962 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 0.
I20260317 21:02:59.183626 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a" is_pre_election: true
I20260317 21:02:59.183713 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 0.
I20260317 21:02:59.183766 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters:
I20260317 21:02:59.183890 30012 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:02:59.183936 30012 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:02:59.183966 30012 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:02:59.184572 30012 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.184742 30012 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 1 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:02:59.184893 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:02:59.184913 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:02:59.184957 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:02:59.184988 29962 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:02:59.185740 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 1.
I20260317 21:02:59.185842 29962 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 1.
I20260317 21:02:59.185978 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:02:59.186084 30012 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:02:59.186259 30012 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 1 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:02:59.186328 30012 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.186968 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 1 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:02:59.198370 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 1 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:02:59.198573 29962 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 1 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:02:59.198659 30027 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:02:59.198798 30012 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:02:59.219592 29677 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20260317 21:02:59.609406 30016 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:02:59.609577 30016 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:02:59.610455 30016 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.610739 30016 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 2 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:02:59.613582 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:02:59.613622 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:02:59.613903 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:02:59.614030 30016 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 21:02:59.614387 30017 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:02:59.614429 30017 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:02:59.615020 30017 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.615203 30017 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 2 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:02:59.618551 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:02:59.618551 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:02:59.618719 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 2: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:02:59.618880 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:02:59.618992 30017 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 21:02:59.622156 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 2 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 1. Current term is 2. Ops: []
I20260317 21:02:59.622465 30027 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 1.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20260317 21:02:59.622629 30027 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260317 21:02:59.622671 30027 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260317 21:02:59.633597 29962 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 2 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 1. Current term is 2. Ops: []
I20260317 21:02:59.633939 30012 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 1.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260317 21:02:59.714800 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 2 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 1. Current term is 2. Ops: []
I20260317 21:02:59.715160 30012 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 1.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260317 21:02:59.715806 30027 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 21:02:59.715893 30027 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:02:59.715965 30027 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.716118 30027 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:02:59.746212 30017 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:02:59.746354 30017 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 2 FOLLOWER]: Advancing to term 3
I20260317 21:02:59.747047 30017 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.747272 30017 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 3 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:02:59.747486 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:02:59.747486 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:02:59.747583 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 2 FOLLOWER]: Advancing to term 3
I20260317 21:02:59.747588 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 2 FOLLOWER]: Advancing to term 3
I20260317 21:02:59.748309 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 3.
I20260317 21:02:59.748309 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 3.
I20260317 21:02:59.748598 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters:
I20260317 21:02:59.748807 30017 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 3 FOLLOWER]: Leader election won for term 3
I20260317 21:02:59.749086 30017 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 3 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:02:59.749200 30017 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.749832 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 1 to 3, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 3 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:02:59.837291 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 3 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20260317 21:02:59.837637 30017 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260317 21:02:59.863047 30016 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:02:59.863183 30016 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 3 FOLLOWER]: Advancing to term 4
I20260317 21:02:59.863914 30016 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.864073 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 4 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 3. Current term is 4. Ops: []
I20260317 21:02:59.864163 30016 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 4 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:02:59.864347 30017 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260317 21:02:59.864346 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 4 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:02:59.864464 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 4 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:02:59.864699 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:02:59.864804 30016 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 21:02:59.897322 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 4 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 3. Current term is 4. Ops: [3.41-3.41]
I20260317 21:02:59.897698 30061 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 1.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260317 21:02:59.909430 30057 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 3 LEADER]: Stepping down as leader of term 3
I20260317 21:02:59.909547 30057 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:02:59.909777 30057 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.910054 30057 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 3 FOLLOWER]: Advancing to term 4
I20260317 21:02:59.941676 30027 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:02:59.941790 30027 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 3 FOLLOWER]: Advancing to term 4
I20260317 21:02:59.942546 30027 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.942739 30027 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 4 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:02:59.943022 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:02:59.943099 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 4: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:02:59.943032 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:02:59.943792 29962 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 4.
I20260317 21:02:59.944039 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:02:59.944136 30027 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 21:02:59.944198 30027 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 4 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:02:59.944270 30027 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.944808 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 3 to 4, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 4 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:02:59.971215 30016 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:02:59.971314 30016 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 4 FOLLOWER]: Advancing to term 5
I20260317 21:02:59.971966 30016 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:02:59.972138 30016 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 5 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:02:59.972415 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 5 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:02:59.972548 29962 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 4 FOLLOWER]: Advancing to term 5
I20260317 21:02:59.972458 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 5 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:02:59.973279 29962 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 5 because replica has last-logged OpId of term: 3 index: 41, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 40.
I20260317 21:02:59.973517 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:02:59.973644 30016 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260317 21:03:00.032433 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 5 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 4. Current term is 5. Ops: []
I20260317 21:03:00.032852 30027 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 42, Last known committed idx: 40, Time since last communication: 0.000s
I20260317 21:03:00.039139 30027 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 4 LEADER]: Stepping down as leader of term 4
I20260317 21:03:00.039269 30027 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:00.039383 30027 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 4.42, Last appended by leader: 42, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.039510 30027 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 4 FOLLOWER]: Advancing to term 5
I20260317 21:03:00.083069 30016 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:00.083179 30016 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 5 FOLLOWER]: Advancing to term 6
I20260317 21:03:00.083847 30016 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.083982 30016 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 6 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:00.084295 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 6 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:00.084376 29962 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 5 FOLLOWER]: Advancing to term 6
I20260317 21:03:00.084308 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 6 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:00.084479 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 5 FOLLOWER]: Advancing to term 6
I20260317 21:03:00.084990 29962 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 6 because replica has last-logged OpId of term: 3 index: 41, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 40.
I20260317 21:03:00.085127 29887 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 6 because replica has last-logged OpId of term: 4 index: 42, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 40.
I20260317 21:03:00.085383 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:00.085481 30016 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260317 21:03:00.135541 30057 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:00.135645 30057 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 6 FOLLOWER]: Advancing to term 7
I20260317 21:03:00.136353 30057 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.136543 30057 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 7 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:00.136785 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 7 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:00.136875 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 6 FOLLOWER]: Advancing to term 7
I20260317 21:03:00.136790 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 7 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:00.136953 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 6 FOLLOWER]: Advancing to term 7
I20260317 21:03:00.137583 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 7.
I20260317 21:03:00.137650 29887 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a for term 7 because replica has last-logged OpId of term: 4 index: 42, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 41.
I20260317 21:03:00.137844 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:03:00.137951 30057 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 7 FOLLOWER]: Leader election won for term 7
I20260317 21:03:00.138028 30057 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 7 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:00.138079 30057 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.138584 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 4 to 7, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 7 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:00.231189 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 7 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 7 index: 42. (term mismatch)
I20260317 21:03:00.231282 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 41
I20260317 21:03:00.231309 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 4.42
I20260317 21:03:00.231335 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:00.231626 30057 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20260317 21:03:00.233081 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 7 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 7 index: 42. (index mismatch)
I20260317 21:03:00.233333 30061 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 40, Time since last communication: 0.000s
I20260317 21:03:00.334046 30027 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:00.334156 30027 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 7 FOLLOWER]: Advancing to term 8
I20260317 21:03:00.334704 30062 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:00.334769 30062 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 7 FOLLOWER]: Advancing to term 8
I20260317 21:03:00.334986 30027 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.335152 30027 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 8 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:00.335379 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 8 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:00.335398 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 8 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:00.335654 30062 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.335759 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 8: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:00.335796 30062 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 8 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:00.335923 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 8 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:00.335930 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 8 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:00.335939 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:00.336035 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 8: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:00.336102 30027 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260317 21:03:00.336163 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:00.336292 30062 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260317 21:03:00.355743 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 8 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 7. Current term is 8. Ops: []
I20260317 21:03:00.356137 30061 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 7.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20260317 21:03:00.358575 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 8 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 7. Current term is 8. Ops: []
I20260317 21:03:00.358875 30057 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 7.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20260317 21:03:00.438438 30061 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 7 LEADER]: Stepping down as leader of term 7
I20260317 21:03:00.438603 30061 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:00.438714 30061 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 7.42, Last appended by leader: 42, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.438869 30061 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 7 FOLLOWER]: Advancing to term 8
I20260317 21:03:00.462919 30062 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:00.463055 30062 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 8 FOLLOWER]: Advancing to term 9
I20260317 21:03:00.463739 30062 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.463927 30062 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 9 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:00.464257 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 9 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:00.464257 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 9 candidate_status { last_received { term: 7 index: 42 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:00.464365 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 8 FOLLOWER]: Advancing to term 9
I20260317 21:03:00.464401 29962 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 8 FOLLOWER]: Advancing to term 9
I20260317 21:03:00.464918 29962 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 9.
I20260317 21:03:00.464965 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 9.
I20260317 21:03:00.465114 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:03:00.465227 30062 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 9 FOLLOWER]: Leader election won for term 9
I20260317 21:03:00.465327 30062 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 9 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:00.465385 30062 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 7.42, Last appended by leader: 42, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.465837 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 7 to 9, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 9 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:00.550107 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 9 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 7 index: 42. Preceding OpId from leader: term: 9 index: 43. (index mismatch)
I20260317 21:03:00.550704 30062 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20260317 21:03:00.577394 29962 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 9 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 7 index: 42. Preceding OpId from leader: term: 9 index: 43. (index mismatch)
I20260317 21:03:00.578032 30062 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20260317 21:03:00.737488 30027 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:00.737602 30027 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 9 FOLLOWER]: Advancing to term 10
I20260317 21:03:00.738293 30027 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.738503 30027 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 10 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:00.738744 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 10 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:00.738761 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 10 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:00.739008 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:00.739168 30027 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260317 21:03:00.743608 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 10 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 9. Current term is 10. Ops: []
I20260317 21:03:00.744029 30063 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 9.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20260317 21:03:00.798334 30063 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 9 LEADER]: Stepping down as leader of term 9
I20260317 21:03:00.798444 30063 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:00.798610 30063 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 9.43, Last appended by leader: 43, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.798722 30063 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 9 FOLLOWER]: Advancing to term 10
I20260317 21:03:00.847707 30027 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:00.847849 30027 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 10 FOLLOWER]: Advancing to term 11
I20260317 21:03:00.848399 30061 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:00.848462 30061 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 9 FOLLOWER]: Advancing to term 10
I20260317 21:03:00.848652 30027 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.848819 30027 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 11 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:00.849085 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 11 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:00.849110 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 11 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:00.849232 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 10 FOLLOWER]: Advancing to term 11
I20260317 21:03:00.849328 30061 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.849479 29962 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 10 FOLLOWER]: Advancing to term 11
I20260317 21:03:00.849588 29810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 10 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:00.849792 30061 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 10 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:00.849988 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 10 candidate_status { last_received { term: 9 index: 43 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:00.850014 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 11.
I20260317 21:03:00.850071 29887 raft_consensus.cc:2368] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 10. Current term is 11.
I20260317 21:03:00.850214 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:03:00.850237 29926 leader_election.cc:400] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 10 election: Vote denied by peer 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653) with higher term. Message: Invalid argument: T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 10. Current term is 11.
I20260317 21:03:00.850267 29926 leader_election.cc:403] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260317 21:03:00.850301 30027 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 11 FOLLOWER]: Leader election won for term 11
I20260317 21:03:00.850391 30027 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 11 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:00.850442 30027 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 9.43, Last appended by leader: 43, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:00.850541 29962 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 11.
I20260317 21:03:00.850602 30061 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653) with higher term. Message: Invalid argument: T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 10. Current term is 11.
I20260317 21:03:00.851007 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 9 to 11, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 11 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:00.857401 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:00.858656 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:00.860800 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:00.864984 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:00.873283 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:00.889564 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:00.921873 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:00.958604 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 11 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 9 index: 43. Preceding OpId from leader: term: 11 index: 44. (index mismatch)
I20260317 21:03:00.959153 30027 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20260317 21:03:00.960436 29962 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 11 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 9 index: 43. Preceding OpId from leader: term: 11 index: 44. (index mismatch)
I20260317 21:03:00.960763 30064 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20260317 21:03:00.986150 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:01.061349 30061 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:01.061471 30061 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 11 FOLLOWER]: Advancing to term 12
I20260317 21:03:01.062320 30061 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.062505 29962 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 12 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 11. Current term is 12. Ops: []
I20260317 21:03:01.062655 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 12 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:01.062711 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 12 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:01.062760 30065 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 11.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20260317 21:03:01.062865 30065 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 11 LEADER]: Stepping down as leader of term 11
I20260317 21:03:01.062508 30061 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 12 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:01.062915 30065 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:01.062857 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:01.062979 30065 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 11.44, Last appended by leader: 44, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.063023 30061 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260317 21:03:01.063056 30065 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 11 FOLLOWER]: Advancing to term 12
I20260317 21:03:01.114466 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:01.153335 30063 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:01.153498 30063 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 11 FOLLOWER]: Advancing to term 12
I20260317 21:03:01.154181 30063 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.154412 30063 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 12 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:01.154704 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 12 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:01.154788 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 12 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:01.154834 29962 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 12: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:01.155452 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 12.
I20260317 21:03:01.155689 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:01.155786 30063 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 12 FOLLOWER]: Leader election won for term 12
I20260317 21:03:01.155834 30063 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 12 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:01.155882 30063 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 11.44, Last appended by leader: 44, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.156373 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 11 to 12, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 12 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:01.188266 30061 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:01.188385 30061 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 12 FOLLOWER]: Advancing to term 13
I20260317 21:03:01.189028 30061 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.189244 30061 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 13 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:01.189495 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 13 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:01.189495 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 13 candidate_status { last_received { term: 11 index: 44 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:01.189641 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 12 FOLLOWER]: Advancing to term 13
I20260317 21:03:01.190270 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 13.
I20260317 21:03:01.190500 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:01.190609 30061 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 13 FOLLOWER]: Leader election won for term 13
I20260317 21:03:01.190750 30061 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 13 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:01.190837 30061 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 11.44, Last appended by leader: 44, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.191349 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 12 to 13, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 13 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:01.263167 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 13 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 12. Current term is 13. Ops: []
I20260317 21:03:01.263582 30063 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20260317 21:03:01.263721 30063 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 12 LEADER]: Stepping down as leader of term 12
I20260317 21:03:01.263749 30063 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:01.263825 30063 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 12.45, Last appended by leader: 45, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.263909 30063 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 12 FOLLOWER]: Advancing to term 13
I20260317 21:03:01.271000 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 13 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 11 index: 44. Preceding OpId from leader: term: 13 index: 45. (index mismatch)
I20260317 21:03:01.271346 30061 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20260317 21:03:01.272758 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 13 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 12 index: 45. Preceding OpId from leader: term: 13 index: 45. (term mismatch)
I20260317 21:03:01.272848 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 44
I20260317 21:03:01.272869 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 12.45
I20260317 21:03:01.272888 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:01.273104 30061 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20260317 21:03:01.370810 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:01.575769 30063 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:01.575891 30063 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 13 FOLLOWER]: Advancing to term 14
I20260317 21:03:01.576628 30063 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.576812 30063 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 14 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:01.577092 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 14 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:01.577135 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 14 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:01.577476 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:01.577597 30063 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260317 21:03:01.595999 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 14 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 13. Current term is 14. Ops: []
I20260317 21:03:01.596334 30067 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 13.45, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20260317 21:03:01.596488 30066 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 13 LEADER]: Stepping down as leader of term 13
I20260317 21:03:01.596549 30066 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:01.596614 30066 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 13.45, Last appended by leader: 45, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.596709 30066 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 13 FOLLOWER]: Advancing to term 14
I20260317 21:03:01.630849 30065 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:01.630965 30065 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 13 FOLLOWER]: Advancing to term 14
I20260317 21:03:01.631657 30065 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.631956 30065 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 14 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:01.632092 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 14 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:01.632166 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 14: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:01.632151 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 14 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:01.632952 29962 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 14.
I20260317 21:03:01.633206 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:01.633308 30065 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 14 FOLLOWER]: Leader election won for term 14
I20260317 21:03:01.633427 30065 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 14 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:01.633497 30065 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 13.45, Last appended by leader: 45, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.633968 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 13 to 14, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 14 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:01.705076 30063 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:01.705202 30063 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 14 FOLLOWER]: Advancing to term 15
I20260317 21:03:01.706024 30063 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.706204 30063 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 15 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:01.706457 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 15 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:01.706533 29962 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 14 FOLLOWER]: Advancing to term 15
I20260317 21:03:01.706518 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 15 candidate_status { last_received { term: 13 index: 45 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:01.707721 29962 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 15.
I20260317 21:03:01.708155 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:01.708314 30063 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 15 FOLLOWER]: Leader election won for term 15
I20260317 21:03:01.708371 30063 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 15 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:01.708441 30063 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 13.45, Last appended by leader: 45, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.709455 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 14 to 15, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 15 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:01.715824 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 15 LEADER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 14. Current term is 15. Ops: []
I20260317 21:03:01.716205 30065 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20260317 21:03:01.716346 30065 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 15.46 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 45
I20260317 21:03:01.716372 30065 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 14 LEADER]: Stepping down as leader of term 14
I20260317 21:03:01.716387 30065 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:01.716473 30065 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 14.46, Last appended by leader: 46, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.716563 30065 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 14 FOLLOWER]: Advancing to term 15
I20260317 21:03:01.797154 29962 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 15 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 13 index: 45. Preceding OpId from leader: term: 15 index: 46. (index mismatch)
I20260317 21:03:01.797484 30063 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20260317 21:03:01.798733 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 15 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 14 index: 46. Preceding OpId from leader: term: 15 index: 46. (term mismatch)
I20260317 21:03:01.798794 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 45
I20260317 21:03:01.798815 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 14.46
I20260317 21:03:01.798833 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:01.798990 30063 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20260317 21:03:01.883057 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:01.899291 30066 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:01.899406 30066 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 15 FOLLOWER]: Advancing to term 16
I20260317 21:03:01.900166 30066 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.900362 30066 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 16 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:01.900658 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 16 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:01.900717 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 16 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:01.900985 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:01.901137 30066 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260317 21:03:01.905402 29962 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 16 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 15. Current term is 16. Ops: []
I20260317 21:03:01.905710 30069 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 15.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20260317 21:03:01.905866 30063 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 15 LEADER]: Stepping down as leader of term 15
I20260317 21:03:01.905910 30063 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:01.905987 30063 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 15.46, Last appended by leader: 46, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.906076 30063 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 15 FOLLOWER]: Advancing to term 16
I20260317 21:03:01.982398 30065 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:01.982514 30065 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 15 FOLLOWER]: Advancing to term 16
I20260317 21:03:01.983227 30065 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.983461 30065 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 16 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:01.983728 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 16 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:01.983754 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 16 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:01.983839 29962 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 16: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:01.984421 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 16.
I20260317 21:03:01.984632 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:01.984750 30065 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 16 FOLLOWER]: Leader election won for term 16
I20260317 21:03:01.984831 30065 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 16 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:01.984890 30065 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 15.46, Last appended by leader: 46, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:01.985374 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 15 to 16, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 16 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:02.019758 30066 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:02.019894 30066 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 16 FOLLOWER]: Advancing to term 17
I20260317 21:03:02.020612 30066 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.020838 30066 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 17 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:02.021117 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 17 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:02.021126 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 17 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:02.021237 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 16 FOLLOWER]: Advancing to term 17
I20260317 21:03:02.021875 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 17.
I20260317 21:03:02.022130 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:02.022236 30066 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 17 FOLLOWER]: Leader election won for term 17
I20260317 21:03:02.022331 30066 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 17 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:02.022388 30066 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 15.46, Last appended by leader: 46, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.022848 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 16 to 17, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 17 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:02.085068 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 17 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 16. Current term is 17. Ops: []
I20260317 21:03:02.085575 30065 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20260317 21:03:02.085688 30065 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 16 LEADER]: Stepping down as leader of term 16
I20260317 21:03:02.085707 30065 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:02.085754 30065 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 16.47, Last appended by leader: 47, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.085839 30065 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 16 FOLLOWER]: Advancing to term 17
I20260317 21:03:02.131585 30063 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:02.131695 30063 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 17 FOLLOWER]: Advancing to term 18
I20260317 21:03:02.132452 30063 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.132632 30063 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 18 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:02.132894 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 18 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:02.132946 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 18 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:02.133046 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 17 FOLLOWER]: Advancing to term 18
I20260317 21:03:02.133684 29887 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 18 because replica has last-logged OpId of term: 16 index: 47, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 46.
I20260317 21:03:02.133919 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:02.134049 30063 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260317 21:03:02.134407 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 18 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 17. Current term is 18. Ops: []
I20260317 21:03:02.134649 30066 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20260317 21:03:02.134752 30066 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 24015e78ad6341a9a2bc13754a55b581 log is divergent from this leader: its last log entry 16.47 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 46
I20260317 21:03:02.134814 30067 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 17 LEADER]: Stepping down as leader of term 17
I20260317 21:03:02.134912 30067 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:02.134982 30067 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 17.47, Last appended by leader: 47, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.135093 30067 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 17 FOLLOWER]: Advancing to term 18
I20260317 21:03:02.282886 30063 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:02.283090 30063 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 18 FOLLOWER]: Advancing to term 19
I20260317 21:03:02.284029 30063 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.284245 30063 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 19 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:02.284672 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 19 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:02.284686 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 19 candidate_status { last_received { term: 15 index: 46 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:02.284765 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 18 FOLLOWER]: Advancing to term 19
I20260317 21:03:02.284826 29962 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 18 FOLLOWER]: Advancing to term 19
I20260317 21:03:02.285458 29887 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 19 because replica has last-logged OpId of term: 16 index: 47, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 46.
I20260317 21:03:02.285575 29962 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 19 because replica has last-logged OpId of term: 17 index: 47, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 46.
I20260317 21:03:02.285773 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:02.285984 30063 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260317 21:03:02.450917 30067 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:02.451028 30067 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 19 FOLLOWER]: Advancing to term 20
I20260317 21:03:02.451733 30067 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.451943 30067 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 20 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:02.452184 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 20 candidate_status { last_received { term: 17 index: 47 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:02.452195 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 20 candidate_status { last_received { term: 17 index: 47 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:02.452273 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 19 FOLLOWER]: Advancing to term 20
I20260317 21:03:02.452312 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 19 FOLLOWER]: Advancing to term 20
I20260317 21:03:02.452903 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 20.
I20260317 21:03:02.452903 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 20.
I20260317 21:03:02.453163 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:03:02.453322 30067 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 20 FOLLOWER]: Leader election won for term 20
I20260317 21:03:02.453387 30067 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 20 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:02.453441 30067 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 17.47, Last appended by leader: 47, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.453930 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:02.530511 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 20 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 16 index: 47. Preceding OpId from leader: term: 20 index: 48. (index mismatch)
I20260317 21:03:02.530858 30067 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 46, Time since last communication: 0.000s
I20260317 21:03:02.530979 30067 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 24015e78ad6341a9a2bc13754a55b581 log is divergent from this leader: its last log entry 16.47 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 46
I20260317 21:03:02.531288 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 46
I20260317 21:03:02.531345 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 16.47
I20260317 21:03:02.531363 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:02.532213 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 20 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 15 index: 46. Preceding OpId from leader: term: 20 index: 48. (index mismatch)
I20260317 21:03:02.532517 30070 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 46, Time since last communication: 0.000s
I20260317 21:03:02.812216 30068 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:02.812390 30068 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 20 FOLLOWER]: Advancing to term 21
I20260317 21:03:02.813103 30068 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.813349 30068 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 21 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:02.813603 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 21 candidate_status { last_received { term: 20 index: 48 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:02.813603 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 21 candidate_status { last_received { term: 20 index: 48 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:02.813855 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:02.813956 30068 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260317 21:03:02.815410 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 21 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 20. Current term is 21. Ops: []
I20260317 21:03:02.815645 30066 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 20.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20260317 21:03:02.815763 30070 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 20 LEADER]: Stepping down as leader of term 20
I20260317 21:03:02.815805 30070 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:02.815847 30070 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 20.48, Last appended by leader: 48, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.815910 30070 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 20 FOLLOWER]: Advancing to term 21
I20260317 21:03:02.826825 30071 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:02.826929 30071 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 20 FOLLOWER]: Advancing to term 21
I20260317 21:03:02.827656 30071 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.827822 30071 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 21 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:02.828060 29962 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 21 candidate_status { last_received { term: 20 index: 48 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:02.828060 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 21 candidate_status { last_received { term: 20 index: 48 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:02.828190 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 21: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:02.828819 29962 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 21.
I20260317 21:03:02.829022 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:02.829136 30071 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 21 FOLLOWER]: Leader election won for term 21
I20260317 21:03:02.829216 30071 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 21 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:02.829270 30071 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 20.48, Last appended by leader: 48, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.829830 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 20 to 21, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 21 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:02.883339 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:02.920378 29962 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 21 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 20 index: 48. Preceding OpId from leader: term: 21 index: 49. (index mismatch)
I20260317 21:03:02.920680 30071 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20260317 21:03:02.920841 30068 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:02.920886 30068 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 21 FOLLOWER]: Advancing to term 22
I20260317 21:03:02.921700 30068 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.921845 30068 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 22 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:02.922031 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 22 candidate_status { last_received { term: 20 index: 48 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:02.922014 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 22 candidate_status { last_received { term: 20 index: 48 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:02.922422 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:02.922433 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 22 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 21. Current term is 22. Ops: []
I20260317 21:03:02.922554 30068 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260317 21:03:02.922693 30073 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20260317 21:03:02.922789 30073 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 21 LEADER]: Stepping down as leader of term 21
I20260317 21:03:02.922818 30073 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:02.922878 30073 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 21.49, Last appended by leader: 49, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:02.922955 30073 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 21 FOLLOWER]: Advancing to term 22
I20260317 21:03:03.041224 30070 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:03.041329 30070 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 21 FOLLOWER]: Advancing to term 22
I20260317 21:03:03.042053 30070 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.042250 30070 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 22 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:03.042479 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 22 candidate_status { last_received { term: 21 index: 49 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:03.042479 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 22 candidate_status { last_received { term: 21 index: 49 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:03.042595 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 22: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:03.043232 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 22.
I20260317 21:03:03.043500 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:03.043640 30070 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 22 FOLLOWER]: Leader election won for term 22
I20260317 21:03:03.043712 30070 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 22 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:03.043766 30070 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 21.49, Last appended by leader: 49, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.044332 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 21 to 22, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 22 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:03.126108 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 22 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 20 index: 48. Preceding OpId from leader: term: 22 index: 50. (index mismatch)
I20260317 21:03:03.126554 30070 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 48, Time since last communication: 0.000s
I20260317 21:03:03.127797 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 22 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 21 index: 49. Preceding OpId from leader: term: 22 index: 50. (index mismatch)
I20260317 21:03:03.128084 30074 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20260317 21:03:03.228173 30068 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:03.228338 30068 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 22 FOLLOWER]: Advancing to term 23
I20260317 21:03:03.229314 30068 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.229532 30068 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 23 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:03.229775 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 23 candidate_status { last_received { term: 22 index: 50 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:03.229781 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 23 candidate_status { last_received { term: 22 index: 50 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:03.230198 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:03.230329 30068 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260317 21:03:03.230435 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 23 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 22. Current term is 23. Ops: []
I20260317 21:03:03.230664 30066 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 22.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20260317 21:03:03.230785 30070 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 22 LEADER]: Stepping down as leader of term 22
I20260317 21:03:03.230825 30070 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:03.230902 30070 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 22.50, Last appended by leader: 50, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.230986 30070 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 22 FOLLOWER]: Advancing to term 23
I20260317 21:03:03.312048 30073 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:03.312160 30073 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 22 FOLLOWER]: Advancing to term 23
I20260317 21:03:03.312861 30073 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.313004 30073 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 23 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:03.313284 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 23 candidate_status { last_received { term: 22 index: 50 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:03.313390 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 23 candidate_status { last_received { term: 22 index: 50 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:03.313517 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 23: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:03.314169 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 23.
I20260317 21:03:03.314376 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:03.314486 30073 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 23 FOLLOWER]: Leader election won for term 23
I20260317 21:03:03.314554 30073 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 23 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:03.314615 30073 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 22.50, Last appended by leader: 50, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.315125 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 22 to 23, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 23 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:03.374333 30068 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:03.374477 30068 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 23 FOLLOWER]: Advancing to term 24
I20260317 21:03:03.375157 30068 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.375387 30068 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 24 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:03.375674 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 24 candidate_status { last_received { term: 22 index: 50 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:03.375717 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 24 candidate_status { last_received { term: 22 index: 50 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:03.375770 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 23 FOLLOWER]: Advancing to term 24
I20260317 21:03:03.376515 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 24.
I20260317 21:03:03.376762 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:03.376935 30068 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 24 FOLLOWER]: Leader election won for term 24
I20260317 21:03:03.377092 30068 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 24 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:03.377231 30068 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 22.50, Last appended by leader: 50, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.377749 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 23 to 24, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 24 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:03.393616 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 24 LEADER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 23. Current term is 24. Ops: []
I20260317 21:03:03.393949 30073 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20260317 21:03:03.394071 30073 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer 24015e78ad6341a9a2bc13754a55b581 log is divergent from this leader: its last log entry 24.51 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 50
I20260317 21:03:03.394140 30072 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 23 LEADER]: Stepping down as leader of term 23
I20260317 21:03:03.394205 30072 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:03.394289 30072 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 23.51, Last appended by leader: 51, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.394397 30072 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 23 FOLLOWER]: Advancing to term 24
I20260317 21:03:03.461612 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 24 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 23 index: 51. Preceding OpId from leader: term: 24 index: 51. (term mismatch)
I20260317 21:03:03.461697 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 50
I20260317 21:03:03.461711 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 23.51
I20260317 21:03:03.461726 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:03.462036 30068 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20260317 21:03:03.463303 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 24 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 22 index: 50. Preceding OpId from leader: term: 24 index: 51. (index mismatch)
I20260317 21:03:03.463608 30076 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20260317 21:03:03.564323 30070 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 24 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:03.564440 30070 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 24 FOLLOWER]: Advancing to term 25
I20260317 21:03:03.565224 30070 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.565421 30070 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 25 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:03.565738 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 25 candidate_status { last_received { term: 24 index: 51 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:03.565738 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 25 candidate_status { last_received { term: 24 index: 51 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:03.565868 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 24 FOLLOWER]: Advancing to term 25
I20260317 21:03:03.566484 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 25.
I20260317 21:03:03.566756 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:03.566874 30070 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 25 FOLLOWER]: Leader election won for term 25
I20260317 21:03:03.566946 30070 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 25 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:03.567000 30070 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 24.51, Last appended by leader: 51, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.567472 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 24 to 25, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 25 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:03.572717 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 25 LEADER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 24. Current term is 25. Ops: []
I20260317 21:03:03.572978 30075 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 24.51, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20260317 21:03:03.573098 30075 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 25.52 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 51
I20260317 21:03:03.573134 30076 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 24 LEADER]: Stepping down as leader of term 24
I20260317 21:03:03.573278 30076 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:03.573359 30076 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 24.51, Last appended by leader: 51, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.573470 30076 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 24 FOLLOWER]: Advancing to term 25
I20260317 21:03:03.656137 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 25 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 24 index: 51. Preceding OpId from leader: term: 25 index: 52. (index mismatch)
I20260317 21:03:03.656531 30070 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20260317 21:03:03.657879 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 25 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 24 index: 51. Preceding OpId from leader: term: 25 index: 52. (index mismatch)
I20260317 21:03:03.658059 30074 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20260317 21:03:03.846818 30072 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:03.846931 30072 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 25 FOLLOWER]: Advancing to term 26
I20260317 21:03:03.847678 30072 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.847857 30072 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 26 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:03.848112 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 26 candidate_status { last_received { term: 25 index: 52 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:03.848173 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 26 candidate_status { last_received { term: 25 index: 52 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:03.848397 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:03.848529 30072 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260317 21:03:03.866134 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 26 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 25. Current term is 26. Ops: []
I20260317 21:03:03.866420 30070 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 25.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20260317 21:03:03.866530 30070 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 25 LEADER]: Stepping down as leader of term 25
I20260317 21:03:03.866549 30070 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:03.866628 30070 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 25.52, Last appended by leader: 52, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.866710 30070 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 25 FOLLOWER]: Advancing to term 26
I20260317 21:03:03.883651 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:03.964787 30072 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:03.964905 30072 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 26 FOLLOWER]: Advancing to term 27
I20260317 21:03:03.965678 30072 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.965821 30072 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 27 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:03.966105 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 27 candidate_status { last_received { term: 25 index: 52 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:03.966140 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 27 candidate_status { last_received { term: 25 index: 52 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:03.966164 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 26 FOLLOWER]: Advancing to term 27
I20260317 21:03:03.966275 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 25 FOLLOWER]: Advancing to term 27
I20260317 21:03:03.966831 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 27.
I20260317 21:03:03.966832 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 27.
I20260317 21:03:03.967254 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:03:03.967383 30072 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 27 FOLLOWER]: Leader election won for term 27
I20260317 21:03:03.967525 30072 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 27 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:03.967590 30072 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 25.52, Last appended by leader: 52, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:03.968084 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 25 to 27, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 27 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:04.043484 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 27 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 25 index: 52. Preceding OpId from leader: term: 27 index: 53. (index mismatch)
I20260317 21:03:04.043828 30072 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20260317 21:03:04.044911 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 27 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 25 index: 52. Preceding OpId from leader: term: 27 index: 53. (index mismatch)
I20260317 21:03:04.045113 30078 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20260317 21:03:04.145632 30070 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:04.145741 30070 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 27 FOLLOWER]: Advancing to term 28
I20260317 21:03:04.146507 30070 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.146726 30070 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 28 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:04.146976 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 28 candidate_status { last_received { term: 27 index: 53 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:04.147053 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 28 candidate_status { last_received { term: 27 index: 53 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:04.147466 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:04.147614 30070 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260317 21:03:04.150760 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 28 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 27. Current term is 28. Ops: []
I20260317 21:03:04.151052 30077 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 27.53, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20260317 21:03:04.151178 30078 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 27 LEADER]: Stepping down as leader of term 27
I20260317 21:03:04.151223 30078 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:04.151321 30078 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 27.53, Last appended by leader: 53, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.151403 30078 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 27 FOLLOWER]: Advancing to term 28
I20260317 21:03:04.223923 30075 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:04.224030 30075 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 27 FOLLOWER]: Advancing to term 28
I20260317 21:03:04.224858 30075 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.225052 30075 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 28 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:04.225337 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 28 candidate_status { last_received { term: 27 index: 53 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:04.225338 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 28 candidate_status { last_received { term: 27 index: 53 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:04.225464 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 28: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:04.225993 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 28.
I20260317 21:03:04.226202 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:04.226305 30075 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 28 FOLLOWER]: Leader election won for term 28
I20260317 21:03:04.226383 30075 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 28 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:04.226454 30075 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 27.53, Last appended by leader: 53, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.227077 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 27 to 28, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 28 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:04.256443 30070 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:04.256558 30070 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 28 FOLLOWER]: Advancing to term 29
I20260317 21:03:04.257210 30070 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.257406 30070 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 29 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:04.257640 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 29 candidate_status { last_received { term: 27 index: 53 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:04.257650 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 29 candidate_status { last_received { term: 27 index: 53 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:04.257706 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 28 FOLLOWER]: Advancing to term 29
I20260317 21:03:04.258342 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 29.
I20260317 21:03:04.258546 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:04.258641 30070 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 29 FOLLOWER]: Leader election won for term 29
I20260317 21:03:04.258692 30070 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 29 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:04.258742 30070 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 27.53, Last appended by leader: 53, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.259217 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 28 to 29, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 29 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:04.302315 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 29 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 28. Current term is 29. Ops: []
I20260317 21:03:04.302739 30075 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20260317 21:03:04.302896 30075 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 28 LEADER]: Stepping down as leader of term 28
I20260317 21:03:04.302932 30075 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:04.302999 30075 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 28.54, Last appended by leader: 54, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.303084 30075 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 28 FOLLOWER]: Advancing to term 29
I20260317 21:03:04.363868 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 29 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 28 index: 54. Preceding OpId from leader: term: 29 index: 54. (term mismatch)
I20260317 21:03:04.363953 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 53
I20260317 21:03:04.363968 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 28.54
I20260317 21:03:04.363982 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:04.364229 30070 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20260317 21:03:04.365566 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 29 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 27 index: 53. Preceding OpId from leader: term: 29 index: 54. (index mismatch)
I20260317 21:03:04.365763 30079 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20260317 21:03:04.551769 30075 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:04.551877 30075 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 29 FOLLOWER]: Advancing to term 30
I20260317 21:03:04.552623 30075 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.552872 30075 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 30 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:04.553136 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 30 candidate_status { last_received { term: 29 index: 54 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:04.553136 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 30 candidate_status { last_received { term: 29 index: 54 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:04.553444 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:04.553556 30075 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260317 21:03:04.555583 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 30 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 29. Current term is 30. Ops: []
I20260317 21:03:04.555871 30074 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 29.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260317 21:03:04.556002 30079 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 29 LEADER]: Stepping down as leader of term 29
I20260317 21:03:04.556041 30079 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:04.556118 30079 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 29.54, Last appended by leader: 54, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.556202 30079 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 29 FOLLOWER]: Advancing to term 30
I20260317 21:03:04.643653 30077 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:04.643750 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 29 FOLLOWER]: Advancing to term 30
I20260317 21:03:04.644484 30077 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.644629 30077 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 30 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:04.644941 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 30 candidate_status { last_received { term: 29 index: 54 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:04.644941 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 30 candidate_status { last_received { term: 29 index: 54 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:04.645099 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 30: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:04.645726 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 30.
I20260317 21:03:04.645915 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:04.646013 30077 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 30 FOLLOWER]: Leader election won for term 30
I20260317 21:03:04.646071 30077 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 30 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:04.646124 30077 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 29.54, Last appended by leader: 54, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.646637 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 29 to 30, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 30 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:04.667192 30075 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:04.667362 30075 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 30 FOLLOWER]: Advancing to term 31
I20260317 21:03:04.668041 30075 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.668267 30075 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 31 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:04.668498 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 31 candidate_status { last_received { term: 29 index: 54 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:04.668557 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 30 FOLLOWER]: Advancing to term 31
I20260317 21:03:04.668552 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 31 candidate_status { last_received { term: 29 index: 54 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:04.669281 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 31.
I20260317 21:03:04.669497 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:04.669595 30075 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 31 FOLLOWER]: Leader election won for term 31
I20260317 21:03:04.669643 30075 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 31 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:04.669735 30075 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 29.54, Last appended by leader: 54, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.670238 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 30 to 31, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 31 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:04.748786 29811 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 30 LEADER]: Stepping down as leader of term 30
I20260317 21:03:04.748929 29811 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:04.749017 29811 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 30.55, Last appended by leader: 55, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:04.749131 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 30 FOLLOWER]: Advancing to term 31
I20260317 21:03:04.750073 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 31 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 30 index: 55. Preceding OpId from leader: term: 31 index: 55. (term mismatch)
I20260317 21:03:04.750118 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 54
I20260317 21:03:04.750131 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 30.55
I20260317 21:03:04.750145 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:04.750519 30075 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260317 21:03:04.751662 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 31 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 29 index: 54. Preceding OpId from leader: term: 31 index: 55. (index mismatch)
I20260317 21:03:04.751852 30076 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260317 21:03:04.883950 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:05.043884 30077 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 31 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:05.043994 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 31 FOLLOWER]: Advancing to term 32
I20260317 21:03:05.044720 30077 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.045013 30077 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 32 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:05.045253 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 32 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:05.045298 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 32 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:05.045524 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:05.045647 30077 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260317 21:03:05.057960 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 32 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 31. Current term is 32. Ops: []
I20260317 21:03:05.058387 30075 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 31.55, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20260317 21:03:05.058655 30080 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 31 LEADER]: Stepping down as leader of term 31
I20260317 21:03:05.058771 30080 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:05.058887 30080 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 31.55, Last appended by leader: 55, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.058998 30080 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 31 FOLLOWER]: Advancing to term 32
I20260317 21:03:05.099711 30079 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 31 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:05.099817 30079 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 31 FOLLOWER]: Advancing to term 32
I20260317 21:03:05.100538 30079 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.100729 30079 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 32 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:05.100942 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 32 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:05.101029 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 32: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:05.100961 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 32 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:05.101662 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 32.
I20260317 21:03:05.101960 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:05.102126 30079 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 32 FOLLOWER]: Leader election won for term 32
I20260317 21:03:05.102370 30079 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 32 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:05.102422 30079 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 31.55, Last appended by leader: 55, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.103037 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 31 to 32, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 32 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:05.178210 30077 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:05.178311 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 32 FOLLOWER]: Advancing to term 33
I20260317 21:03:05.178962 30077 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.179126 30077 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 33 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:05.179389 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 33 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:05.179389 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 33 candidate_status { last_received { term: 31 index: 55 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:05.179498 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 32 FOLLOWER]: Advancing to term 33
I20260317 21:03:05.180065 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 33.
I20260317 21:03:05.180308 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:05.180398 30077 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 33 FOLLOWER]: Leader election won for term 33
I20260317 21:03:05.180505 30077 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 33 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:05.180546 30077 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 31.55, Last appended by leader: 55, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.181092 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 32 to 33, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 33 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:05.216589 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 33 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 32. Current term is 33. Ops: []
I20260317 21:03:05.217032 30079 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20260317 21:03:05.217152 30079 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 32 LEADER]: Stepping down as leader of term 32
I20260317 21:03:05.217196 30079 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:05.217283 30079 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 32.56, Last appended by leader: 56, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.217358 30079 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 32 FOLLOWER]: Advancing to term 33
I20260317 21:03:05.256608 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 33 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 32 index: 56. Preceding OpId from leader: term: 33 index: 56. (term mismatch)
I20260317 21:03:05.256708 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 55
I20260317 21:03:05.256724 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 32.56
I20260317 21:03:05.256742 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:05.257064 30077 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20260317 21:03:05.258419 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 33 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 31 index: 55. Preceding OpId from leader: term: 33 index: 56. (index mismatch)
I20260317 21:03:05.258903 30077 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20260317 21:03:05.554260 30079 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:05.554384 30079 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 33 FOLLOWER]: Advancing to term 34
I20260317 21:03:05.555981 30079 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.556198 30079 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 34 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:05.556409 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 34 candidate_status { last_received { term: 33 index: 56 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:05.556459 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 34 candidate_status { last_received { term: 33 index: 56 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:05.556689 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:05.556777 30079 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260317 21:03:05.569280 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 34 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 33. Current term is 34. Ops: []
I20260317 21:03:05.569643 30083 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 33.56, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20260317 21:03:05.569792 30077 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 33 LEADER]: Stepping down as leader of term 33
I20260317 21:03:05.569839 30077 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:05.569885 30077 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 33.56, Last appended by leader: 56, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.569955 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 33 FOLLOWER]: Advancing to term 34
I20260317 21:03:05.639595 30075 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:05.639735 30075 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 33 FOLLOWER]: Advancing to term 34
I20260317 21:03:05.640424 30075 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.640623 30075 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 34 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:05.640834 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 34 candidate_status { last_received { term: 33 index: 56 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:05.640913 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 34: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:05.640839 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 34 candidate_status { last_received { term: 33 index: 56 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:05.641666 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 34.
I20260317 21:03:05.641888 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:05.642005 30075 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 34 FOLLOWER]: Leader election won for term 34
I20260317 21:03:05.642092 30075 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 34 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:05.642153 30075 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 33.56, Last appended by leader: 56, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.642597 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 33 to 34, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 34 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:05.684767 30079 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:05.684881 30079 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 34 FOLLOWER]: Advancing to term 35
I20260317 21:03:05.685940 30079 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.686153 30079 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 35 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:05.686394 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 35 candidate_status { last_received { term: 33 index: 56 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:05.686390 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 35 candidate_status { last_received { term: 33 index: 56 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:05.686553 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 34 FOLLOWER]: Advancing to term 35
I20260317 21:03:05.687125 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 35.
I20260317 21:03:05.687376 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:05.687520 30079 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 35 FOLLOWER]: Leader election won for term 35
I20260317 21:03:05.687579 30079 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 35 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:05.687635 30079 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 33.56, Last appended by leader: 56, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.688169 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 34 to 35, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 35 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:05.723547 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 35 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 34. Current term is 35. Ops: []
I20260317 21:03:05.723875 30075 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20260317 21:03:05.724027 30075 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 34 LEADER]: Stepping down as leader of term 34
I20260317 21:03:05.724057 30075 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:05.724121 30075 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 34.57, Last appended by leader: 57, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.724223 30075 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 34 FOLLOWER]: Advancing to term 35
I20260317 21:03:05.783854 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 35 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 33 index: 56. Preceding OpId from leader: term: 35 index: 57. (index mismatch)
I20260317 21:03:05.784133 30079 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20260317 21:03:05.785694 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 35 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 34 index: 57. Preceding OpId from leader: term: 35 index: 57. (term mismatch)
I20260317 21:03:05.785771 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 56
I20260317 21:03:05.785797 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 34.57
I20260317 21:03:05.785830 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:05.786156 30081 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20260317 21:03:05.885775 30077 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 35 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:05.885890 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 35 FOLLOWER]: Advancing to term 36
I20260317 21:03:05.886615 30077 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.886792 30077 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 36 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:05.887053 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 36 candidate_status { last_received { term: 35 index: 57 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:05.887053 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 36 candidate_status { last_received { term: 35 index: 57 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:05.887184 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 35 FOLLOWER]: Advancing to term 36
I20260317 21:03:05.887758 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 36.
I20260317 21:03:05.887967 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:05.888085 30077 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 36 FOLLOWER]: Leader election won for term 36
I20260317 21:03:05.888165 30077 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 36 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:05.888226 30077 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 35.57, Last appended by leader: 57, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.888806 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:05.888890 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 36 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 35. Current term is 36. Ops: []
I20260317 21:03:05.889063 30081 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 35.57, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20260317 21:03:05.889017 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 35 to 36, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 36 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:05.889199 30084 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 35 LEADER]: Stepping down as leader of term 35
I20260317 21:03:05.889252 30084 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:05.889326 30084 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 35.57, Last appended by leader: 57, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:05.889405 30084 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 35 FOLLOWER]: Advancing to term 36
I20260317 21:03:05.979457 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 36 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 35 index: 57. Preceding OpId from leader: term: 36 index: 58. (index mismatch)
I20260317 21:03:05.979926 30077 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20260317 21:03:05.981608 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 36 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 35 index: 57. Preceding OpId from leader: term: 36 index: 58. (index mismatch)
I20260317 21:03:05.982020 30085 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20260317 21:03:06.081962 30075 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:06.082086 30075 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 36 FOLLOWER]: Advancing to term 37
I20260317 21:03:06.082764 30075 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.082988 30075 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 37 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:06.083211 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 37 candidate_status { last_received { term: 36 index: 58 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:06.083211 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 37 candidate_status { last_received { term: 36 index: 58 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:06.083331 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 36 FOLLOWER]: Advancing to term 37
I20260317 21:03:06.083865 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 37.
I20260317 21:03:06.084105 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:06.084206 30075 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 37 FOLLOWER]: Leader election won for term 37
I20260317 21:03:06.084311 30075 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 37 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:06.084370 30075 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 36.58, Last appended by leader: 58, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.084935 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 36 to 37, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 37 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:06.086423 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 37 LEADER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 36. Current term is 37. Ops: []
I20260317 21:03:06.086690 30085 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 36.58, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20260317 21:03:06.086795 30085 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer 24015e78ad6341a9a2bc13754a55b581 log is divergent from this leader: its last log entry 37.59 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 58
I20260317 21:03:06.086833 30077 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 36 LEADER]: Stepping down as leader of term 36
I20260317 21:03:06.086879 30077 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:06.086947 30077 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 36.58, Last appended by leader: 58, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.087021 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 36 FOLLOWER]: Advancing to term 37
I20260317 21:03:06.176456 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 37 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 36 index: 58. Preceding OpId from leader: term: 37 index: 59. (index mismatch)
I20260317 21:03:06.176934 30075 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20260317 21:03:06.178143 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 37 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 36 index: 58. Preceding OpId from leader: term: 37 index: 59. (index mismatch)
I20260317 21:03:06.178361 30086 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20260317 21:03:06.279740 30084 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 37 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:06.279845 30084 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 37 FOLLOWER]: Advancing to term 38
I20260317 21:03:06.280548 30084 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.280741 30084 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 38 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:06.280988 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 38 candidate_status { last_received { term: 37 index: 59 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:06.280997 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 38 candidate_status { last_received { term: 37 index: 59 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:06.281360 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:06.281483 30084 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260317 21:03:06.286814 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 38 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 37. Current term is 38. Ops: []
I20260317 21:03:06.287077 30075 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 37.59, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20260317 21:03:06.287180 30075 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 37 LEADER]: Stepping down as leader of term 37
I20260317 21:03:06.287199 30075 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:06.287317 30075 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 37.59, Last appended by leader: 59, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.287400 30075 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 37 FOLLOWER]: Advancing to term 38
I20260317 21:03:06.361981 30077 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 37 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:06.362136 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 37 FOLLOWER]: Advancing to term 38
I20260317 21:03:06.362850 30077 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.363059 30077 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 38 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:06.363277 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 38 candidate_status { last_received { term: 37 index: 59 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:06.363361 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 38: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:06.363359 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 38 candidate_status { last_received { term: 37 index: 59 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:06.364161 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 38.
I20260317 21:03:06.364403 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:06.364521 30077 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 38 FOLLOWER]: Leader election won for term 38
I20260317 21:03:06.364589 30077 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 38 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:06.364647 30077 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 37.59, Last appended by leader: 59, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.365214 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 37 to 38, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 38 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:06.430600 30084 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:06.430755 30084 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 38 FOLLOWER]: Advancing to term 39
I20260317 21:03:06.431479 30084 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.431700 30084 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 39 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:06.431954 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 39 candidate_status { last_received { term: 37 index: 59 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:06.432037 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 38 FOLLOWER]: Advancing to term 39
I20260317 21:03:06.431964 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 39 candidate_status { last_received { term: 37 index: 59 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:06.432669 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 39.
I20260317 21:03:06.432881 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:06.432981 30084 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 39 FOLLOWER]: Leader election won for term 39
I20260317 21:03:06.433105 30084 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 39 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:06.433162 30084 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 37.59, Last appended by leader: 59, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.433789 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 38 to 39, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 39 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:06.471987 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 39 LEADER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 38. Current term is 39. Ops: []
I20260317 21:03:06.472416 30077 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20260317 21:03:06.472535 30077 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 39.60 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 59
I20260317 21:03:06.472561 30077 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 38 LEADER]: Stepping down as leader of term 38
I20260317 21:03:06.472576 30077 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:06.472644 30077 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 38.60, Last appended by leader: 60, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.472723 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 38 FOLLOWER]: Advancing to term 39
I20260317 21:03:06.533469 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 39 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 38 index: 60. Preceding OpId from leader: term: 39 index: 60. (term mismatch)
I20260317 21:03:06.533563 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 59
I20260317 21:03:06.533578 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 38.60
I20260317 21:03:06.533592 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:06.533851 30084 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20260317 21:03:06.535046 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 39 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 37 index: 59. Preceding OpId from leader: term: 39 index: 60. (index mismatch)
I20260317 21:03:06.535298 30084 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20260317 21:03:06.636250 30086 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:06.636374 30086 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 39 FOLLOWER]: Advancing to term 40
I20260317 21:03:06.637521 30086 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.637735 30086 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 40 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:06.637967 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 40 candidate_status { last_received { term: 39 index: 60 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:06.638018 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 40 candidate_status { last_received { term: 39 index: 60 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:06.638052 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 39 FOLLOWER]: Advancing to term 40
I20260317 21:03:06.638957 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 40.
I20260317 21:03:06.639164 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:06.639286 30086 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 40 FOLLOWER]: Leader election won for term 40
I20260317 21:03:06.639369 30086 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 40 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:06.639415 30086 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 39.60, Last appended by leader: 60, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.639923 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 39 to 40, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 40 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:06.642843 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 40 LEADER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 39. Current term is 40. Ops: []
I20260317 21:03:06.643132 30088 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 39.60, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20260317 21:03:06.643260 30088 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 24015e78ad6341a9a2bc13754a55b581 log is divergent from this leader: its last log entry 40.61 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 60
I20260317 21:03:06.643296 30087 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 39 LEADER]: Stepping down as leader of term 39
I20260317 21:03:06.643350 30087 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:06.643420 30087 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 39.60, Last appended by leader: 60, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.643515 30087 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 39 FOLLOWER]: Advancing to term 40
I20260317 21:03:06.749545 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 40 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 39 index: 60. Preceding OpId from leader: term: 40 index: 61. (index mismatch)
I20260317 21:03:06.749902 30086 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20260317 21:03:06.751039 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 40 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 39 index: 60. Preceding OpId from leader: term: 40 index: 61. (index mismatch)
I20260317 21:03:06.751235 30090 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20260317 21:03:06.851797 30077 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 40 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:06.851912 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 40 FOLLOWER]: Advancing to term 41
I20260317 21:03:06.852536 30077 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.852720 30077 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 41 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:06.852989 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 41 candidate_status { last_received { term: 40 index: 61 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:06.853030 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 41 candidate_status { last_received { term: 40 index: 61 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:06.853047 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 40 FOLLOWER]: Advancing to term 41
I20260317 21:03:06.853824 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 41.
I20260317 21:03:06.854027 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:06.854125 30077 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 41 FOLLOWER]: Leader election won for term 41
I20260317 21:03:06.854174 30077 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 41 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:06.854231 30077 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 40.61, Last appended by leader: 61, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.854712 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 40 to 41, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 41 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:06.863982 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 41 LEADER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 40. Current term is 41. Ops: []
I20260317 21:03:06.864375 30090 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 40.61, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20260317 21:03:06.864522 30090 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 41.62 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 61
I20260317 21:03:06.864598 30089 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 40 LEADER]: Stepping down as leader of term 40
I20260317 21:03:06.864672 30089 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:06.864748 30089 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 40.61, Last appended by leader: 61, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:06.864849 30089 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 40 FOLLOWER]: Advancing to term 41
I20260317 21:03:06.889077 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:06.939179 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 41 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 40 index: 61. Preceding OpId from leader: term: 41 index: 62. (index mismatch)
I20260317 21:03:06.939486 30077 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20260317 21:03:06.940892 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 41 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 40 index: 61. Preceding OpId from leader: term: 41 index: 62. (index mismatch)
I20260317 21:03:06.941256 30085 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20260317 21:03:07.121654 30087 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:07.121760 30087 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 41 FOLLOWER]: Advancing to term 42
I20260317 21:03:07.122473 30087 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.122677 30087 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 42 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:07.122926 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:07.122926 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:07.123168 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:07.123337 30087 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260317 21:03:07.136045 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 42 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 41. Current term is 42. Ops: []
I20260317 21:03:07.136366 30077 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 41.62, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20260317 21:03:07.136473 30077 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 41 LEADER]: Stepping down as leader of term 41
I20260317 21:03:07.136490 30077 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:07.136569 30077 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 41.62, Last appended by leader: 62, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.136646 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 41 FOLLOWER]: Advancing to term 42
I20260317 21:03:07.163477 30089 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:07.163585 30089 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 41 FOLLOWER]: Advancing to term 42
I20260317 21:03:07.164296 30089 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.164513 30089 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 42 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:07.164803 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:07.164875 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 42: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:07.164816 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:07.165766 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 42.
I20260317 21:03:07.166076 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:07.166270 30089 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 42 FOLLOWER]: Leader election won for term 42
I20260317 21:03:07.166363 30089 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 42 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:07.166419 30089 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 41.62, Last appended by leader: 62, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.167023 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 41 to 42, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 42 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:07.245479 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 42 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 41 index: 62. Preceding OpId from leader: term: 42 index: 63. (index mismatch)
I20260317 21:03:07.245839 30089 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20260317 21:03:07.247035 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 42 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 41 index: 62. Preceding OpId from leader: term: 42 index: 63. (index mismatch)
I20260317 21:03:07.247233 30086 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20260317 21:03:07.348551 30091 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 42 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:07.348706 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 42 FOLLOWER]: Advancing to term 43
I20260317 21:03:07.349541 30091 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.349769 30091 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 43 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:07.350013 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 43 candidate_status { last_received { term: 42 index: 63 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:07.350059 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 43 candidate_status { last_received { term: 42 index: 63 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:07.350356 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:07.350448 30091 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260317 21:03:07.362941 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 43 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 42. Current term is 43. Ops: []
I20260317 21:03:07.363309 30086 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 42.63, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20260317 21:03:07.363543 30090 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 42 LEADER]: Stepping down as leader of term 42
I20260317 21:03:07.363592 30090 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:07.363658 30090 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 42.63, Last appended by leader: 63, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.363724 30090 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 42 FOLLOWER]: Advancing to term 43
I20260317 21:03:07.445660 30077 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 42 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:07.445775 30077 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 42 FOLLOWER]: Advancing to term 43
I20260317 21:03:07.446506 30077 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.446674 30077 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 43 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:07.446923 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 43 candidate_status { last_received { term: 42 index: 63 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:07.446965 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 43 candidate_status { last_received { term: 42 index: 63 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:07.447031 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 43: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:07.447576 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 43.
I20260317 21:03:07.447783 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:07.447897 30077 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 43 FOLLOWER]: Leader election won for term 43
I20260317 21:03:07.447952 30077 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 43 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:07.447997 30077 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 42.63, Last appended by leader: 63, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.448894 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 42 to 43, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 43 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:07.452836 30091 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:07.452919 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 43 FOLLOWER]: Advancing to term 44
I20260317 21:03:07.453691 30091 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.453984 30091 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 44 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:07.454133 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 44 candidate_status { last_received { term: 42 index: 63 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:07.454198 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 44 candidate_status { last_received { term: 42 index: 63 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:07.454244 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 43 FOLLOWER]: Advancing to term 44
I20260317 21:03:07.454900 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 44.
I20260317 21:03:07.455143 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:07.455263 30091 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 44 FOLLOWER]: Leader election won for term 44
I20260317 21:03:07.455355 30091 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 44 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:07.455410 30091 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 42.63, Last appended by leader: 63, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.455945 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 43 to 44, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 44 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:07.536068 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 44 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 42 index: 63. Preceding OpId from leader: term: 44 index: 64. (index mismatch)
I20260317 21:03:07.536477 30091 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20260317 21:03:07.537850 29811 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 43 LEADER]: Stepping down as leader of term 43
I20260317 21:03:07.537907 29811 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:07.537966 29811 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 43.64, Last appended by leader: 64, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.538048 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 43 FOLLOWER]: Advancing to term 44
I20260317 21:03:07.538839 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 44 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 43 index: 64. Preceding OpId from leader: term: 44 index: 64. (term mismatch)
I20260317 21:03:07.538869 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 63
I20260317 21:03:07.538882 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 43.64
I20260317 21:03:07.538894 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:07.539148 30092 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20260317 21:03:07.639074 30090 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:07.639222 30090 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 44 FOLLOWER]: Advancing to term 45
I20260317 21:03:07.639942 30090 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.640166 30090 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 45 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:07.640538 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 45 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:07.640538 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 45 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:07.640688 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 44 FOLLOWER]: Advancing to term 45
I20260317 21:03:07.641366 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 45.
I20260317 21:03:07.641593 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:07.641706 30090 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 45 FOLLOWER]: Leader election won for term 45
I20260317 21:03:07.641793 30090 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 45 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:07.641860 30090 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 44.64, Last appended by leader: 64, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.642114 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 45 LEADER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 44. Current term is 45. Ops: []
I20260317 21:03:07.642351 30092 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 44.64, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20260317 21:03:07.642400 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 44 to 45, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 45 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:07.642458 30092 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 24015e78ad6341a9a2bc13754a55b581 log is divergent from this leader: its last log entry 45.65 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 64
I20260317 21:03:07.642531 30091 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 44 LEADER]: Stepping down as leader of term 44
I20260317 21:03:07.642565 30091 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:07.642616 30091 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 44.64, Last appended by leader: 64, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.642695 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 44 FOLLOWER]: Advancing to term 45
I20260317 21:03:07.720165 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 45 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 44 index: 64. Preceding OpId from leader: term: 45 index: 65. (index mismatch)
I20260317 21:03:07.720518 30090 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20260317 21:03:07.721707 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 45 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 44 index: 64. Preceding OpId from leader: term: 45 index: 65. (index mismatch)
I20260317 21:03:07.721901 30089 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20260317 21:03:07.889351 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:07.899305 30085 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 45 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:07.899415 30085 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 45 FOLLOWER]: Advancing to term 46
I20260317 21:03:07.900252 30085 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.900511 30085 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 46 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:07.900753 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 46 candidate_status { last_received { term: 45 index: 65 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:07.900753 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 46 candidate_status { last_received { term: 45 index: 65 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:07.901021 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:07.901139 30085 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260317 21:03:07.923058 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 46 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 45. Current term is 46. Ops: []
I20260317 21:03:07.923403 30090 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 45.65, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20260317 21:03:07.923556 30090 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 45 LEADER]: Stepping down as leader of term 45
I20260317 21:03:07.923611 30090 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:07.923714 30090 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 45.65, Last appended by leader: 65, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.923817 30090 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 45 FOLLOWER]: Advancing to term 46
I20260317 21:03:07.987866 30091 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 45 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:07.987969 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 45 FOLLOWER]: Advancing to term 46
I20260317 21:03:07.988677 30091 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.988862 30091 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 46 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:07.989086 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 46 candidate_status { last_received { term: 45 index: 65 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:07.989212 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 46 candidate_status { last_received { term: 45 index: 65 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:07.989313 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 46: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:07.989847 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 46.
I20260317 21:03:07.990068 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:07.990168 30091 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 46 FOLLOWER]: Leader election won for term 46
I20260317 21:03:07.990217 30091 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 46 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:07.990276 30091 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 45.65, Last appended by leader: 65, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:07.990927 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 45 to 46, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 46 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:08.043699 30085 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:08.043834 30085 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 46 FOLLOWER]: Advancing to term 47
I20260317 21:03:08.044620 30085 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.044821 30085 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 47 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:08.045081 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 47 candidate_status { last_received { term: 45 index: 65 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:08.045171 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 46 FOLLOWER]: Advancing to term 47
I20260317 21:03:08.045091 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 47 candidate_status { last_received { term: 45 index: 65 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:08.045876 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 47.
I20260317 21:03:08.046160 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:08.046329 30085 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 47 FOLLOWER]: Leader election won for term 47
I20260317 21:03:08.046499 30085 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 47 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:08.046584 30085 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 45.65, Last appended by leader: 65, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.047143 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 46 to 47, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 47 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:08.082340 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 47 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 46. Current term is 47. Ops: []
I20260317 21:03:08.082719 30091 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20260317 21:03:08.082860 30091 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 46 LEADER]: Stepping down as leader of term 46
I20260317 21:03:08.082894 30091 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:08.082960 30091 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 46.66, Last appended by leader: 66, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.083035 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 46 FOLLOWER]: Advancing to term 47
I20260317 21:03:08.141621 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 47 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 46 index: 66. Preceding OpId from leader: term: 47 index: 66. (term mismatch)
I20260317 21:03:08.141706 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 65
I20260317 21:03:08.141722 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 46.66
I20260317 21:03:08.141738 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:08.142083 30085 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20260317 21:03:08.143355 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 47 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 45 index: 65. Preceding OpId from leader: term: 47 index: 66. (index mismatch)
I20260317 21:03:08.143621 30085 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20260317 21:03:08.244863 30089 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:08.245019 30089 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 47 FOLLOWER]: Advancing to term 48
I20260317 21:03:08.245819 30089 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.246074 30089 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 48 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:08.246285 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 48 candidate_status { last_received { term: 47 index: 66 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:08.246340 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 48 candidate_status { last_received { term: 47 index: 66 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:08.246560 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:08.246610 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 48 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 47. Current term is 48. Ops: []
I20260317 21:03:08.246670 30089 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260317 21:03:08.246901 30093 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 47.66, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20260317 21:03:08.247018 30093 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 47 LEADER]: Stepping down as leader of term 47
I20260317 21:03:08.247056 30093 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:08.247138 30093 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 47.66, Last appended by leader: 66, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.247226 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 47 FOLLOWER]: Advancing to term 48
I20260317 21:03:08.324026 30091 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 47 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:08.324177 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 47 FOLLOWER]: Advancing to term 48
I20260317 21:03:08.324885 30091 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.325119 30091 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 48 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:08.325392 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 48 candidate_status { last_received { term: 47 index: 66 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:08.325398 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 48 candidate_status { last_received { term: 47 index: 66 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:08.325498 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 48: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:08.326148 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 48.
I20260317 21:03:08.326368 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:08.326538 30091 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 48 FOLLOWER]: Leader election won for term 48
I20260317 21:03:08.326617 30091 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 48 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:08.326654 30091 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 47.66, Last appended by leader: 66, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.327203 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 47 to 48, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 48 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:08.357962 30089 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:08.358070 30089 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 48 FOLLOWER]: Advancing to term 49
I20260317 21:03:08.358757 30089 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.358952 30089 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 49 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:08.359179 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 49 candidate_status { last_received { term: 47 index: 66 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:08.359186 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 49 candidate_status { last_received { term: 47 index: 66 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:08.359335 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 48 FOLLOWER]: Advancing to term 49
I20260317 21:03:08.359951 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 49.
I20260317 21:03:08.360153 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:08.360308 30089 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 49 FOLLOWER]: Leader election won for term 49
I20260317 21:03:08.360368 30089 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 49 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:08.360423 30089 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 47.66, Last appended by leader: 66, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.361001 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 48 to 49, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 49 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:08.404626 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 49 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 48. Current term is 49. Ops: []
I20260317 21:03:08.404944 30091 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20260317 21:03:08.405054 30091 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 48 LEADER]: Stepping down as leader of term 48
I20260317 21:03:08.405076 30091 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:08.405148 30091 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 48.67, Last appended by leader: 67, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.405316 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 48 FOLLOWER]: Advancing to term 49
I20260317 21:03:08.472602 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 49 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 47 index: 66. Preceding OpId from leader: term: 49 index: 67. (index mismatch)
I20260317 21:03:08.472927 30089 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20260317 21:03:08.474038 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 49 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 48 index: 67. Preceding OpId from leader: term: 49 index: 67. (term mismatch)
I20260317 21:03:08.474097 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 66
I20260317 21:03:08.474123 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 48.67
I20260317 21:03:08.474146 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:08.474303 30094 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20260317 21:03:08.656057 30093 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 49 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:08.656173 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 49 FOLLOWER]: Advancing to term 50
I20260317 21:03:08.656967 30093 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.657212 30093 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 50 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:08.657546 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 50 candidate_status { last_received { term: 49 index: 67 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:08.657546 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 50 candidate_status { last_received { term: 49 index: 67 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:08.657835 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:08.657949 30093 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260317 21:03:08.661296 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 50 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 49. Current term is 50. Ops: []
I20260317 21:03:08.661624 30094 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 49.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20260317 21:03:08.661798 30094 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 49 LEADER]: Stepping down as leader of term 49
I20260317 21:03:08.661857 30094 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:08.661934 30094 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 49.67, Last appended by leader: 67, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.662029 30094 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 49 FOLLOWER]: Advancing to term 50
I20260317 21:03:08.697095 30091 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 49 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:08.697232 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 49 FOLLOWER]: Advancing to term 50
I20260317 21:03:08.697888 30091 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.698097 30091 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 50 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:08.698355 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 50 candidate_status { last_received { term: 49 index: 67 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:08.698382 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 50 candidate_status { last_received { term: 49 index: 67 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:08.698525 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 50: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:08.699029 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 50.
I20260317 21:03:08.699286 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:08.699404 30091 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 50 FOLLOWER]: Leader election won for term 50
I20260317 21:03:08.699482 30091 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 50 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:08.699556 30091 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 49.67, Last appended by leader: 67, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.700044 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 49 to 50, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 50 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:08.789860 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 50 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 49 index: 67. Preceding OpId from leader: term: 50 index: 68. (index mismatch)
I20260317 21:03:08.790256 30091 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20260317 21:03:08.791621 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 50 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 49 index: 67. Preceding OpId from leader: term: 50 index: 68. (index mismatch)
I20260317 21:03:08.791836 30091 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20260317 21:03:08.889633 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:08.893049 30096 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 50 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:08.893133 30096 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 50 FOLLOWER]: Advancing to term 51
I20260317 21:03:08.894306 30096 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.894479 30096 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 51 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:08.894711 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 51 candidate_status { last_received { term: 50 index: 68 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:08.894790 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 51 candidate_status { last_received { term: 50 index: 68 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:08.894999 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:08.895097 30096 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260317 21:03:08.901705 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 51 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 50. Current term is 51. Ops: []
I20260317 21:03:08.902002 30091 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 50.68, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20260317 21:03:08.902122 30091 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 50 LEADER]: Stepping down as leader of term 50
I20260317 21:03:08.902164 30091 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:08.902220 30091 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 50.68, Last appended by leader: 68, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.902294 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 50 FOLLOWER]: Advancing to term 51
I20260317 21:03:08.980067 30094 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 50 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:08.980167 30094 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 50 FOLLOWER]: Advancing to term 51
I20260317 21:03:08.980846 30094 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.981022 30094 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 51 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:08.981323 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 51 candidate_status { last_received { term: 50 index: 68 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:08.981323 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 51 candidate_status { last_received { term: 50 index: 68 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:08.981515 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 51: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:08.982121 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 51.
I20260317 21:03:08.982319 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:08.982427 30094 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 51 FOLLOWER]: Leader election won for term 51
I20260317 21:03:08.982507 30094 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 51 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:08.982585 30094 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 50.68, Last appended by leader: 68, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:08.983186 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 50 to 51, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 51 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:09.001524 30096 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:09.001634 30096 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 51 FOLLOWER]: Advancing to term 52
I20260317 21:03:09.002424 30096 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.002600 30096 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 52 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:09.002832 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 52 candidate_status { last_received { term: 50 index: 68 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:09.002890 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 52 candidate_status { last_received { term: 50 index: 68 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:09.002972 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 51 FOLLOWER]: Advancing to term 52
I20260317 21:03:09.003569 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 52.
I20260317 21:03:09.003837 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:09.004006 30096 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 52 FOLLOWER]: Leader election won for term 52
I20260317 21:03:09.004101 30096 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 52 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:09.004150 30096 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 50.68, Last appended by leader: 68, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.004750 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 51 to 52, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 52 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:09.065587 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 52 LEADER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 51. Current term is 52. Ops: []
I20260317 21:03:09.065913 30094 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20260317 21:03:09.066043 30094 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 52.69 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 68
I20260317 21:03:09.066090 30089 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 51 LEADER]: Stepping down as leader of term 51
I20260317 21:03:09.066143 30089 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:09.066227 30089 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 51.69, Last appended by leader: 69, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.066321 30089 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 51 FOLLOWER]: Advancing to term 52
I20260317 21:03:09.089290 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 52 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 51 index: 69. Preceding OpId from leader: term: 52 index: 69. (term mismatch)
I20260317 21:03:09.089394 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 68
I20260317 21:03:09.089416 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 51.69
I20260317 21:03:09.089439 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 52 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:09.089702 30096 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20260317 21:03:09.090785 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 52 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 50 index: 68. Preceding OpId from leader: term: 52 index: 69. (index mismatch)
I20260317 21:03:09.091003 30093 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20260317 21:03:09.378434 30091 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 52 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:09.378526 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 52 FOLLOWER]: Advancing to term 53
I20260317 21:03:09.379158 30091 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.379339 30091 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 53 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:09.379577 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 53 candidate_status { last_received { term: 52 index: 69 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:09.379585 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 53 candidate_status { last_received { term: 52 index: 69 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:09.379855 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:09.379967 30091 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20260317 21:03:09.400385 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 53 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 52. Current term is 53. Ops: []
I20260317 21:03:09.400770 30097 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 52.69, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20260317 21:03:09.400931 30093 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 52 LEADER]: Stepping down as leader of term 52
I20260317 21:03:09.400976 30093 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:09.401046 30093 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 52.69, Last appended by leader: 69, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.401121 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 52 FOLLOWER]: Advancing to term 53
I20260317 21:03:09.484432 30089 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 52 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:09.484558 30089 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 52 FOLLOWER]: Advancing to term 53
I20260317 21:03:09.485390 30089 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.485584 30089 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 53 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:09.485841 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 53 candidate_status { last_received { term: 52 index: 69 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:09.485874 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 53 candidate_status { last_received { term: 52 index: 69 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:09.485992 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 53: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:09.486521 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 53.
I20260317 21:03:09.486716 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:09.486855 30089 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 53 FOLLOWER]: Leader election won for term 53
I20260317 21:03:09.486937 30089 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 53 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:09.486974 30089 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 52.69, Last appended by leader: 69, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.487396 30091 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:09.487459 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 53 FOLLOWER]: Advancing to term 54
I20260317 21:03:09.487728 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 52 to 53, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 53 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:09.488108 30091 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.488457 30091 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 54 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:09.488422 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 54 candidate_status { last_received { term: 52 index: 69 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:09.488472 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 54 candidate_status { last_received { term: 52 index: 69 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:09.488649 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 53 FOLLOWER]: Advancing to term 54
I20260317 21:03:09.489334 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 54.
I20260317 21:03:09.489573 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:09.489684 30091 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 54 FOLLOWER]: Leader election won for term 54
I20260317 21:03:09.489769 30091 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 54 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:09.489822 30091 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 52.69, Last appended by leader: 69, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.490373 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 53 to 54, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 54 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:09.566949 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 54 LEADER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 53. Current term is 54. Ops: []
I20260317 21:03:09.567265 30089 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20260317 21:03:09.567401 30089 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 54.70 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69
I20260317 21:03:09.567451 30094 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 53 LEADER]: Stepping down as leader of term 53
I20260317 21:03:09.567498 30094 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:09.567601 30094 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 53.70, Last appended by leader: 70, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.567672 30094 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 53 FOLLOWER]: Advancing to term 54
I20260317 21:03:09.593910 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 54 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 52 index: 69. Preceding OpId from leader: term: 54 index: 70. (index mismatch)
I20260317 21:03:09.594262 30091 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20260317 21:03:09.595292 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 54 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 53 index: 70. Preceding OpId from leader: term: 54 index: 70. (term mismatch)
I20260317 21:03:09.595367 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 69
I20260317 21:03:09.595388 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 53.70
I20260317 21:03:09.595412 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 54 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:09.595643 30091 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20260317 21:03:09.696363 30093 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 54 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:09.696516 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 54 FOLLOWER]: Advancing to term 55
I20260317 21:03:09.697309 30093 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.697510 30093 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 55 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:09.697800 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 55 candidate_status { last_received { term: 54 index: 70 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:09.697943 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 55 candidate_status { last_received { term: 54 index: 70 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:09.698048 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 54 FOLLOWER]: Advancing to term 55
I20260317 21:03:09.698711 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 55.
I20260317 21:03:09.698945 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:09.699061 30093 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 55 FOLLOWER]: Leader election won for term 55
I20260317 21:03:09.699139 30093 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 55 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:09.699195 30093 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 54.70, Last appended by leader: 70, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.699713 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 54 to 55, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 55 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:09.703336 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 55 LEADER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 54. Current term is 55. Ops: []
I20260317 21:03:09.703744 30091 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 54.70, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20260317 21:03:09.703975 30091 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 55.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 70
I20260317 21:03:09.704042 30098 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 54 LEADER]: Stepping down as leader of term 54
I20260317 21:03:09.704113 30098 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:09.704258 30098 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 54.70, Last appended by leader: 70, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.704370 30098 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 54 FOLLOWER]: Advancing to term 55
I20260317 21:03:09.787277 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 55 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 54 index: 70. Preceding OpId from leader: term: 55 index: 71. (index mismatch)
I20260317 21:03:09.787649 30093 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20260317 21:03:09.788771 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 55 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 54 index: 70. Preceding OpId from leader: term: 55 index: 71. (index mismatch)
I20260317 21:03:09.789036 30093 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20260317 21:03:09.889909 30094 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 55 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:09.889900 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:09.890045 30094 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 55 FOLLOWER]: Advancing to term 56
I20260317 21:03:09.890740 30094 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.890936 30094 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 56 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:09.891120 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 56 candidate_status { last_received { term: 55 index: 71 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:09.891135 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 56 candidate_status { last_received { term: 55 index: 71 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:09.891397 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:09.891492 30094 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20260317 21:03:09.907590 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 56 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 55. Current term is 56. Ops: []
I20260317 21:03:09.907881 30093 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 55.71, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20260317 21:03:09.908017 30093 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 55 LEADER]: Stepping down as leader of term 55
I20260317 21:03:09.908053 30093 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 55 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:09.908103 30093 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 55.71, Last appended by leader: 71, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.908159 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 55 FOLLOWER]: Advancing to term 56
I20260317 21:03:09.990860 30098 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 55 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:09.990969 30098 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 55 FOLLOWER]: Advancing to term 56
I20260317 21:03:09.991649 30098 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.991869 30098 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 56 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:09.992091 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 56 candidate_status { last_received { term: 55 index: 71 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:09.992091 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 56 candidate_status { last_received { term: 55 index: 71 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:09.992204 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 56: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:09.992738 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 56.
I20260317 21:03:09.992929 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:09.993033 30098 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 56 FOLLOWER]: Leader election won for term 56
I20260317 21:03:09.993109 30098 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 56 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:09.993165 30098 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 55.71, Last appended by leader: 71, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:09.993639 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 55 to 56, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 56 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:10.003211 30094 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 56 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:10.003336 30094 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 56 FOLLOWER]: Advancing to term 57
I20260317 21:03:10.004141 30094 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.004444 30094 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 57 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:10.004550 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 57 candidate_status { last_received { term: 55 index: 71 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:10.004550 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 57 candidate_status { last_received { term: 55 index: 71 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:10.004699 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 56 FOLLOWER]: Advancing to term 57
I20260317 21:03:10.005565 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 57.
I20260317 21:03:10.005774 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:10.005880 30094 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 57 FOLLOWER]: Leader election won for term 57
I20260317 21:03:10.005947 30094 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 57 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:10.006011 30094 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 55.71, Last appended by leader: 71, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.006520 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 56 to 57, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 57 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:10.108703 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 57 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 56. Current term is 57. Ops: []
I20260317 21:03:10.108989 30098 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20260317 21:03:10.109095 30098 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 56 LEADER]: Stepping down as leader of term 56
I20260317 21:03:10.109113 30098 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 56 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:10.109227 30098 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 56.72, Last appended by leader: 72, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.109313 30098 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 56 FOLLOWER]: Advancing to term 57
I20260317 21:03:10.119243 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 57 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 56 index: 72. Preceding OpId from leader: term: 57 index: 72. (term mismatch)
I20260317 21:03:10.119268 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 57 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 55 index: 71. Preceding OpId from leader: term: 57 index: 72. (index mismatch)
I20260317 21:03:10.119349 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 71
I20260317 21:03:10.119371 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 56.72
I20260317 21:03:10.119395 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 57 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:10.119532 30089 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20260317 21:03:10.119642 30094 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20260317 21:03:10.420241 30098 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 57 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:10.420368 30098 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 57 FOLLOWER]: Advancing to term 58
I20260317 21:03:10.421015 30098 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.421259 30098 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 58 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:10.421478 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 58 candidate_status { last_received { term: 57 index: 72 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:10.421484 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 58 candidate_status { last_received { term: 57 index: 72 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:10.421736 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 58 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:10.421861 30098 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 58 FOLLOWER]: Leader election lost for term 58. Reason: could not achieve majority
I20260317 21:03:10.434100 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 58 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 57. Current term is 58. Ops: []
I20260317 21:03:10.434556 30101 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 57.72, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20260317 21:03:10.434690 30101 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 57 LEADER]: Stepping down as leader of term 57
I20260317 21:03:10.434710 30101 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 57 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:10.434787 30101 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 57.72, Last appended by leader: 72, Current term: 57, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.434868 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 57 FOLLOWER]: Advancing to term 58
I20260317 21:03:10.498078 30093 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 57 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:10.498193 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 57 FOLLOWER]: Advancing to term 58
I20260317 21:03:10.498935 30093 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.499112 30093 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 58 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:10.499366 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 58 candidate_status { last_received { term: 57 index: 72 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:10.499476 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 58: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:10.499404 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 58 candidate_status { last_received { term: 57 index: 72 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:10.500063 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 58.
I20260317 21:03:10.500285 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:10.500396 30093 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 58 FOLLOWER]: Leader election won for term 58
I20260317 21:03:10.500500 30093 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 58 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:10.500571 30093 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 57.72, Last appended by leader: 72, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.501094 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 57 to 58, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 58 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:10.539031 30098 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 58 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:10.539139 30098 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 58 FOLLOWER]: Advancing to term 59
I20260317 21:03:10.539830 30098 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.540020 30098 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 59 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:10.540244 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 59 candidate_status { last_received { term: 57 index: 72 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:10.540306 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 58 FOLLOWER]: Advancing to term 59
I20260317 21:03:10.540256 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 59 candidate_status { last_received { term: 57 index: 72 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:10.540982 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 59.
I20260317 21:03:10.541244 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:10.541347 30098 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 59 FOLLOWER]: Leader election won for term 59
I20260317 21:03:10.541424 30098 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 59 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:10.541477 30098 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 57.72, Last appended by leader: 72, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.542344 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 58 to 59, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 59 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:10.578727 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 59 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 58. Current term is 59. Ops: []
I20260317 21:03:10.578982 30093 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20260317 21:03:10.579092 30093 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 58 LEADER]: Stepping down as leader of term 58
I20260317 21:03:10.579110 30093 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 58 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:10.579180 30093 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 58.73, Last appended by leader: 73, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.579259 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 58 FOLLOWER]: Advancing to term 59
I20260317 21:03:10.631596 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 59 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 57 index: 72. Preceding OpId from leader: term: 59 index: 73. (index mismatch)
I20260317 21:03:10.631989 30098 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20260317 21:03:10.633932 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 59 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 58 index: 73. Preceding OpId from leader: term: 59 index: 73. (term mismatch)
I20260317 21:03:10.634019 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 72
I20260317 21:03:10.634044 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 58.73
I20260317 21:03:10.634063 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 59 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:10.634402 30091 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20260317 21:03:10.890291 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:10.926600 30101 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 59 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:10.926721 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 59 FOLLOWER]: Advancing to term 60
I20260317 21:03:10.927456 30101 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.927680 30101 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 60 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:10.927906 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 60 candidate_status { last_received { term: 59 index: 73 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:10.927928 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 60 candidate_status { last_received { term: 59 index: 73 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:10.928203 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:10.928349 30101 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I20260317 21:03:10.940146 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 60 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 59. Current term is 60. Ops: []
I20260317 21:03:10.940492 30098 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 59.73, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20260317 21:03:10.940603 30098 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 59 LEADER]: Stepping down as leader of term 59
I20260317 21:03:10.940621 30098 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 59 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:10.940707 30098 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 59.73, Last appended by leader: 73, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.940812 30098 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 59 FOLLOWER]: Advancing to term 60
I20260317 21:03:10.993651 30093 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 59 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:10.993759 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 59 FOLLOWER]: Advancing to term 60
I20260317 21:03:10.994534 30093 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.994684 30093 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 60 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:10.995035 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 60 candidate_status { last_received { term: 59 index: 73 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:10.995035 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 60 candidate_status { last_received { term: 59 index: 73 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:10.995196 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 60: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:10.995800 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 60 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 60.
I20260317 21:03:10.996006 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 60 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:10.996116 30093 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 60 FOLLOWER]: Leader election won for term 60
I20260317 21:03:10.996268 30093 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 60 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:10.996417 30093 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 59.73, Last appended by leader: 73, Current term: 60, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:10.997061 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 59 to 60, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 60 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:11.038584 30101 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 60 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:11.038690 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 60 FOLLOWER]: Advancing to term 61
I20260317 21:03:11.039431 30101 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.039618 30101 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 61 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:11.039930 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 61 candidate_status { last_received { term: 59 index: 73 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:11.039933 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 61 candidate_status { last_received { term: 59 index: 73 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:11.040073 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 60 FOLLOWER]: Advancing to term 61
I20260317 21:03:11.040644 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 61.
I20260317 21:03:11.040861 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 61 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:11.040964 30101 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 61 FOLLOWER]: Leader election won for term 61
I20260317 21:03:11.041080 30101 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 61 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:11.041122 30101 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 59.73, Last appended by leader: 73, Current term: 61, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.041685 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 60 to 61, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 61 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:11.088601 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 61 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 60. Current term is 61. Ops: []
I20260317 21:03:11.088938 30093 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20260317 21:03:11.089085 30093 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 60 LEADER]: Stepping down as leader of term 60
I20260317 21:03:11.089121 30093 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 60 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:11.089215 30093 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 60.74, Last appended by leader: 74, Current term: 60, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.089335 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 60 FOLLOWER]: Advancing to term 61
I20260317 21:03:11.137373 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 61 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 60 index: 74. Preceding OpId from leader: term: 61 index: 74. (term mismatch)
I20260317 21:03:11.137496 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 73
I20260317 21:03:11.137521 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 60.74
I20260317 21:03:11.137544 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 61 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:11.137851 30101 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20260317 21:03:11.138990 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 61 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 59 index: 73. Preceding OpId from leader: term: 61 index: 74. (index mismatch)
I20260317 21:03:11.139185 30101 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20260317 21:03:11.240545 30098 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 61 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:11.240654 30098 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 61 FOLLOWER]: Advancing to term 62
I20260317 21:03:11.241480 30098 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.241669 30098 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 62 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:11.241906 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 62 candidate_status { last_received { term: 61 index: 74 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:11.241917 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 62 candidate_status { last_received { term: 61 index: 74 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:11.241967 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 61 FOLLOWER]: Advancing to term 62
I20260317 21:03:11.242652 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 62 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 62.
I20260317 21:03:11.242856 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 62 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:11.242969 30098 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 62 FOLLOWER]: Leader election won for term 62
I20260317 21:03:11.243041 30098 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 62 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:11.243117 30098 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 61.74, Last appended by leader: 74, Current term: 62, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.243614 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 61 to 62, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 62 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:11.245337 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 62 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 61. Current term is 62. Ops: []
I20260317 21:03:11.245563 30101 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 61.74, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20260317 21:03:11.245664 30101 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 61 LEADER]: Stepping down as leader of term 61
I20260317 21:03:11.245699 30101 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 61 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:11.245770 30101 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 61.74, Last appended by leader: 74, Current term: 61, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.245851 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 61 FOLLOWER]: Advancing to term 62
I20260317 21:03:11.322280 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 62 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 61 index: 74. Preceding OpId from leader: term: 62 index: 75. (index mismatch)
I20260317 21:03:11.322640 30098 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20260317 21:03:11.323650 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 62 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 61 index: 74. Preceding OpId from leader: term: 62 index: 75. (index mismatch)
I20260317 21:03:11.323863 30098 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20260317 21:03:11.424103 30093 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 62 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:11.424250 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 62 FOLLOWER]: Advancing to term 63
I20260317 21:03:11.424957 30093 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.425149 30093 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 63 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:11.425424 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 63 candidate_status { last_received { term: 62 index: 75 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:11.425495 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 63 candidate_status { last_received { term: 62 index: 75 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:11.425738 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 63 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:11.425848 30093 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 63 FOLLOWER]: Leader election lost for term 63. Reason: could not achieve majority
I20260317 21:03:11.444370 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 63 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 62. Current term is 63. Ops: []
I20260317 21:03:11.444742 30091 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 62.75, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260317 21:03:11.444878 30091 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 62 LEADER]: Stepping down as leader of term 62
I20260317 21:03:11.444936 30091 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 62 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:11.444998 30091 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 62.75, Last appended by leader: 75, Current term: 62, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.445080 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 62 FOLLOWER]: Advancing to term 63
I20260317 21:03:11.503096 30101 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 62 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:11.503214 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 62 FOLLOWER]: Advancing to term 63
I20260317 21:03:11.503888 30101 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.504138 30101 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 63 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:11.504444 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 63 candidate_status { last_received { term: 62 index: 75 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:11.504444 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 63 candidate_status { last_received { term: 62 index: 75 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:11.504580 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 63 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 63: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:11.505110 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 63.
I20260317 21:03:11.505335 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 63 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:11.505467 30101 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 63 FOLLOWER]: Leader election won for term 63
I20260317 21:03:11.505532 30101 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 63 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:11.505606 30101 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 62.75, Last appended by leader: 75, Current term: 63, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.506057 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 62 to 63, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 63 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:11.560146 30093 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 63 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:11.560329 30093 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 63 FOLLOWER]: Advancing to term 64
I20260317 21:03:11.561028 30093 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.561323 30093 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 64 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:11.561578 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 64 candidate_status { last_received { term: 62 index: 75 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:11.561609 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 64 candidate_status { last_received { term: 62 index: 75 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:11.561663 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 63 FOLLOWER]: Advancing to term 64
I20260317 21:03:11.562343 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 64 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 64.
I20260317 21:03:11.562574 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 64 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:11.562709 30093 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 64 FOLLOWER]: Leader election won for term 64
I20260317 21:03:11.562814 30093 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 64 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:11.562871 30093 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 62.75, Last appended by leader: 75, Current term: 64, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.563373 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 63 to 64, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 64 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:11.593685 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 64 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 63. Current term is 64. Ops: []
I20260317 21:03:11.594034 30101 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260317 21:03:11.594151 30101 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 63 LEADER]: Stepping down as leader of term 63
I20260317 21:03:11.594172 30101 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 63 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:11.594218 30101 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 63.76, Last appended by leader: 76, Current term: 63, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.594295 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 63 FOLLOWER]: Advancing to term 64
I20260317 21:03:11.651130 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 64 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 63 index: 76. Preceding OpId from leader: term: 64 index: 76. (term mismatch)
I20260317 21:03:11.651216 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 75
I20260317 21:03:11.651232 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 63.76
I20260317 21:03:11.651252 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 64 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:11.651594 30093 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260317 21:03:11.653019 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 64 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 62 index: 75. Preceding OpId from leader: term: 64 index: 76. (index mismatch)
I20260317 21:03:11.653371 30105 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260317 21:03:11.839064 30091 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 64 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:11.839188 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 64 FOLLOWER]: Advancing to term 65
I20260317 21:03:11.839829 30091 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.840039 30091 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 65 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:11.840245 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 65 candidate_status { last_received { term: 64 index: 76 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:11.840270 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 65 candidate_status { last_received { term: 64 index: 76 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:11.840548 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 65 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:11.840651 30091 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 65 FOLLOWER]: Leader election lost for term 65. Reason: could not achieve majority
I20260317 21:03:11.850898 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 65 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 64. Current term is 65. Ops: []
I20260317 21:03:11.851228 30102 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 64.76, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20260317 21:03:11.851409 30105 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 64 LEADER]: Stepping down as leader of term 64
I20260317 21:03:11.851459 30105 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 64 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:11.851548 30105 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 64.76, Last appended by leader: 76, Current term: 64, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.851644 30105 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 64 FOLLOWER]: Advancing to term 65
I20260317 21:03:11.863121 30101 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 64 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:11.863245 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 64 FOLLOWER]: Advancing to term 65
I20260317 21:03:11.863905 30101 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.864116 30101 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 65 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:11.864316 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 65 candidate_status { last_received { term: 64 index: 76 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:11.864411 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 65 candidate_status { last_received { term: 64 index: 76 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:11.864533 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 65 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 65: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:11.865036 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 65 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 65.
I20260317 21:03:11.865335 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 65 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:11.865432 30101 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 65 FOLLOWER]: Leader election won for term 65
I20260317 21:03:11.865506 30101 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 65 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:11.865594 30101 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 64.76, Last appended by leader: 76, Current term: 65, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.866149 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 64 to 65, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 65 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:11.890614 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:11.946538 30091 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 65 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:11.946651 30091 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 65 FOLLOWER]: Advancing to term 66
I20260317 21:03:11.947320 30091 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.947506 30091 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 66 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:11.947681 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 66 candidate_status { last_received { term: 64 index: 76 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:11.947773 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 66 candidate_status { last_received { term: 64 index: 76 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:11.947780 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 65 FOLLOWER]: Advancing to term 66
I20260317 21:03:11.948343 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 66 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 66.
I20260317 21:03:11.948545 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 66 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:11.948717 30091 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 66 FOLLOWER]: Leader election won for term 66
I20260317 21:03:11.948901 30091 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 66 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:11.948974 30091 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 64.76, Last appended by leader: 76, Current term: 66, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.949463 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 65 to 66, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 66 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:11.976738 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 66 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 65. Current term is 66. Ops: []
I20260317 21:03:11.977087 30101 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20260317 21:03:11.977253 30101 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 65 LEADER]: Stepping down as leader of term 65
I20260317 21:03:11.977275 30101 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 65 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:11.977361 30101 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 65.77, Last appended by leader: 77, Current term: 65, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:11.977449 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 65 FOLLOWER]: Advancing to term 66
I20260317 21:03:12.034525 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 66 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 65 index: 77. Preceding OpId from leader: term: 66 index: 77. (term mismatch)
I20260317 21:03:12.034610 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 76
I20260317 21:03:12.034624 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 65.77
I20260317 21:03:12.034639 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 66 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:12.034881 30091 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20260317 21:03:12.036257 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 66 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 64 index: 76. Preceding OpId from leader: term: 66 index: 77. (index mismatch)
I20260317 21:03:12.036473 30107 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20260317 21:03:12.232575 30105 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 66 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:12.232683 30105 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 66 FOLLOWER]: Advancing to term 67
I20260317 21:03:12.233525 30105 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.233704 30105 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 67 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:12.233987 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 67 candidate_status { last_received { term: 66 index: 77 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:12.234021 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 67 candidate_status { last_received { term: 66 index: 77 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:12.234241 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 67 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:12.234346 30105 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 67 FOLLOWER]: Leader election lost for term 67. Reason: could not achieve majority
I20260317 21:03:12.235396 30101 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 66 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:12.235462 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 66 FOLLOWER]: Advancing to term 67
I20260317 21:03:12.236117 30101 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.236286 30101 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 67 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:12.236506 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 67 candidate_status { last_received { term: 66 index: 77 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:12.236505 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 67 candidate_status { last_received { term: 66 index: 77 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:12.236589 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 67 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 67: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:12.236740 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 67 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:12.236817 30101 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 67 FOLLOWER]: Leader election lost for term 67. Reason: could not achieve majority
I20260317 21:03:12.245391 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 67 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 66. Current term is 67. Ops: []
I20260317 21:03:12.245783 30091 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 66.77, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20260317 21:03:12.245940 30106 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 66 LEADER]: Stepping down as leader of term 66
I20260317 21:03:12.245986 30106 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 66 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:12.246068 30106 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 66.77, Last appended by leader: 77, Current term: 66, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.246156 30106 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 66 FOLLOWER]: Advancing to term 67
I20260317 21:03:12.361034 30105 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 67 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:12.361156 30105 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 67 FOLLOWER]: Advancing to term 68
I20260317 21:03:12.361908 30105 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.362094 30105 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 68 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:12.362373 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 68 candidate_status { last_received { term: 66 index: 77 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:12.362461 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 67 FOLLOWER]: Advancing to term 68
I20260317 21:03:12.362393 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 68 candidate_status { last_received { term: 66 index: 77 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:12.362498 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 67 FOLLOWER]: Advancing to term 68
I20260317 21:03:12.363020 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 68 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 68.
I20260317 21:03:12.363042 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 68 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 68.
I20260317 21:03:12.363293 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 68 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:03:12.363396 30105 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 68 FOLLOWER]: Leader election won for term 68
I20260317 21:03:12.363512 30105 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 68 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:12.363554 30105 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 66.77, Last appended by leader: 77, Current term: 68, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.364135 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 66 to 68, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 68 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:12.471616 30106 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 68 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:12.471715 30106 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 68 FOLLOWER]: Advancing to term 69
I20260317 21:03:12.472385 30106 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.472585 30106 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 69 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:12.472805 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 69 candidate_status { last_received { term: 66 index: 77 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:12.472817 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 69 candidate_status { last_received { term: 66 index: 77 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:12.472909 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 68 FOLLOWER]: Advancing to term 69
I20260317 21:03:12.473600 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 69 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 69.
I20260317 21:03:12.473809 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 69 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:12.473912 30106 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 69 FOLLOWER]: Leader election won for term 69
I20260317 21:03:12.473977 30106 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 69 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:12.474072 30106 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 66.77, Last appended by leader: 77, Current term: 69, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.474550 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 68 to 69, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 69 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:12.486805 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 69 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 68. Current term is 69. Ops: []
I20260317 21:03:12.487165 30108 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20260317 21:03:12.487360 30108 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 68 LEADER]: Stepping down as leader of term 68
I20260317 21:03:12.487421 30108 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 68 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:12.487489 30108 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 68.78, Last appended by leader: 78, Current term: 68, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.487574 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 69 LEADER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 68. Current term is 69. Ops: []
I20260317 21:03:12.487600 30108 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 68 FOLLOWER]: Advancing to term 69
I20260317 21:03:12.561260 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 69 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 66 index: 77. Preceding OpId from leader: term: 69 index: 78. (index mismatch)
I20260317 21:03:12.561539 30106 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20260317 21:03:12.563196 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 69 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 68 index: 78. Preceding OpId from leader: term: 69 index: 78. (term mismatch)
I20260317 21:03:12.563252 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 77
I20260317 21:03:12.563275 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 68.78
I20260317 21:03:12.563297 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 69 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:12.563450 30107 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20260317 21:03:12.786378 30108 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 69 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:12.786480 30108 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 69 FOLLOWER]: Advancing to term 70
I20260317 21:03:12.787165 30108 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.787353 30108 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 70 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:12.787606 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 70 candidate_status { last_received { term: 69 index: 78 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:12.787628 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 70 candidate_status { last_received { term: 69 index: 78 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:12.787839 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 70 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:12.787947 30108 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 70 FOLLOWER]: Leader election lost for term 70. Reason: could not achieve majority
I20260317 21:03:12.796764 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 70 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 69. Current term is 70. Ops: []
I20260317 21:03:12.797014 30107 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 69.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20260317 21:03:12.797117 30107 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 69 LEADER]: Stepping down as leader of term 69
I20260317 21:03:12.797135 30107 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 69 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:12.797227 30107 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 69.78, Last appended by leader: 78, Current term: 69, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.797324 30107 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 69 FOLLOWER]: Advancing to term 70
I20260317 21:03:12.852303 30101 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 69 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:12.852452 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 69 FOLLOWER]: Advancing to term 70
I20260317 21:03:12.853116 30101 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.853384 30101 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 70 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:12.853636 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 70 candidate_status { last_received { term: 69 index: 78 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:12.853641 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 70 candidate_status { last_received { term: 69 index: 78 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:12.853813 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 70 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 70: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:12.854272 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 70 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 70.
I20260317 21:03:12.854470 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 70 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:12.854583 30101 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 70 FOLLOWER]: Leader election won for term 70
I20260317 21:03:12.854684 30101 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 70 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:12.854750 30101 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 69.78, Last appended by leader: 78, Current term: 70, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.855252 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 69 to 70, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 70 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:12.890915 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:12.905359 30108 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 70 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:12.905539 30108 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 70 FOLLOWER]: Advancing to term 71
I20260317 21:03:12.906280 30108 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.906461 30108 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 71 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:12.906699 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 71 candidate_status { last_received { term: 69 index: 78 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:12.906775 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 71 candidate_status { last_received { term: 69 index: 78 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:12.906814 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 70 FOLLOWER]: Advancing to term 71
I20260317 21:03:12.907568 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 71 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 71.
I20260317 21:03:12.907821 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 71 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:12.907928 30108 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 71 FOLLOWER]: Leader election won for term 71
I20260317 21:03:12.908001 30108 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 71 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:12.908049 30108 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 69.78, Last appended by leader: 78, Current term: 71, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.908572 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 70 to 71, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 71 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:12.954908 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 71 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 70. Current term is 71. Ops: []
I20260317 21:03:12.955189 30101 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20260317 21:03:12.955338 30101 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 70 LEADER]: Stepping down as leader of term 70
I20260317 21:03:12.955396 30101 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 70 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:12.955456 30101 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 70.79, Last appended by leader: 79, Current term: 70, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:12.955564 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 70 FOLLOWER]: Advancing to term 71
I20260317 21:03:12.997326 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 71 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 70 index: 79. Preceding OpId from leader: term: 71 index: 79. (term mismatch)
I20260317 21:03:12.997351 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 71 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 69 index: 78. Preceding OpId from leader: term: 71 index: 79. (index mismatch)
I20260317 21:03:12.997452 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 78
I20260317 21:03:12.997494 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 70.79
I20260317 21:03:12.997529 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 71 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:12.997735 30108 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20260317 21:03:12.997834 30105 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20260317 21:03:13.099402 30107 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 71 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:13.099517 30107 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 71 FOLLOWER]: Advancing to term 72
I20260317 21:03:13.100176 30107 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.100386 30107 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 72 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:13.100628 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 72 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:13.100628 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 72 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:13.100965 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 72 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:13.101114 30107 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 72 FOLLOWER]: Leader election lost for term 72. Reason: could not achieve majority
I20260317 21:03:13.106700 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 72 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 71. Current term is 72. Ops: []
I20260317 21:03:13.106988 30105 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 71.79, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20260317 21:03:13.107101 30105 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 71 LEADER]: Stepping down as leader of term 71
I20260317 21:03:13.107120 30105 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 71 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:13.107197 30105 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 71.79, Last appended by leader: 79, Current term: 71, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.107303 30105 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 71 FOLLOWER]: Advancing to term 72
I20260317 21:03:13.191188 30101 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 71 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:13.191306 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 71 FOLLOWER]: Advancing to term 72
I20260317 21:03:13.192070 30101 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.192278 30101 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 72 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:13.192507 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 72 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:13.192507 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 72 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:13.192637 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 72 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 72: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:13.193222 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 72 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 72.
I20260317 21:03:13.193431 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 72 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:13.193536 30101 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 72 FOLLOWER]: Leader election won for term 72
I20260317 21:03:13.193626 30101 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 72 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:13.193679 30101 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 71.79, Last appended by leader: 79, Current term: 72, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.194139 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 71 to 72, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 72 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:13.225329 30107 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 72 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:13.225438 30107 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 72 FOLLOWER]: Advancing to term 73
I20260317 21:03:13.226203 30107 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.226394 30107 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 73 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:13.226609 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 73 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:13.226730 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 72 FOLLOWER]: Advancing to term 73
I20260317 21:03:13.226652 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 73 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:13.227351 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 73.
I20260317 21:03:13.227651 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 73 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:13.227756 30107 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 73 FOLLOWER]: Leader election won for term 73
I20260317 21:03:13.227839 30107 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 73 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:13.227893 30107 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 71.79, Last appended by leader: 79, Current term: 73, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.228392 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 72 to 73, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 73 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:13.312863 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 73 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 72. Current term is 73. Ops: []
I20260317 21:03:13.313287 30101 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20260317 21:03:13.313423 30101 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 72 LEADER]: Stepping down as leader of term 72
I20260317 21:03:13.313478 30101 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 72 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:13.313555 30101 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 72.80, Last appended by leader: 80, Current term: 72, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.313632 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 72 FOLLOWER]: Advancing to term 73
I20260317 21:03:13.338990 30105 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 73 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:13.339114 30105 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 73 FOLLOWER]: Advancing to term 74
I20260317 21:03:13.339761 30105 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.339895 30105 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 74 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:13.340193 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 74 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:13.340234 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 74 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:13.340353 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 73 FOLLOWER]: Advancing to term 74
I20260317 21:03:13.341274 29887 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 74 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 74 because replica has last-logged OpId of term: 72 index: 80, which is greater than that of the candidate, which has last-logged OpId of term: 71 index: 79.
I20260317 21:03:13.341549 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 74 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:13.341671 30105 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 74 FOLLOWER]: Leader election lost for term 74. Reason: could not achieve majority
I20260317 21:03:13.343907 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 74 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 73. Current term is 74. Ops: []
I20260317 21:03:13.344252 30107 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20260317 21:03:13.344384 30107 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 73 LEADER]: Stepping down as leader of term 73
I20260317 21:03:13.344425 30107 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 73 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:13.344487 30107 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 73.80, Last appended by leader: 80, Current term: 73, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.344570 30107 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 73 FOLLOWER]: Advancing to term 74
I20260317 21:03:13.460302 30105 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 74 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:13.460417 30105 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 74 FOLLOWER]: Advancing to term 75
I20260317 21:03:13.461227 30105 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.461416 30105 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 75 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:13.461670 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 75 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:13.461722 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 75 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:13.461787 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 74 FOLLOWER]: Advancing to term 75
I20260317 21:03:13.461813 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 74 FOLLOWER]: Advancing to term 75
I20260317 21:03:13.462368 29887 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 75 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 75 because replica has last-logged OpId of term: 72 index: 80, which is greater than that of the candidate, which has last-logged OpId of term: 71 index: 79.
I20260317 21:03:13.462385 29961 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 75 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 75 because replica has last-logged OpId of term: 73 index: 80, which is greater than that of the candidate, which has last-logged OpId of term: 71 index: 79.
I20260317 21:03:13.462613 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 75 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:13.462738 30105 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 75 FOLLOWER]: Leader election lost for term 75. Reason: could not achieve majority
I20260317 21:03:13.645279 30105 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 75 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:13.645380 30105 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 75 FOLLOWER]: Advancing to term 76
I20260317 21:03:13.646095 30105 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.646234 30105 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 76 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:13.646551 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 76 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:13.646608 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 76 candidate_status { last_received { term: 71 index: 79 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:13.646651 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 75 FOLLOWER]: Advancing to term 76
I20260317 21:03:13.646658 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 75 FOLLOWER]: Advancing to term 76
I20260317 21:03:13.647282 29887 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 76 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 76 because replica has last-logged OpId of term: 72 index: 80, which is greater than that of the candidate, which has last-logged OpId of term: 71 index: 79.
I20260317 21:03:13.647320 29961 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 76 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 76 because replica has last-logged OpId of term: 73 index: 80, which is greater than that of the candidate, which has last-logged OpId of term: 71 index: 79.
I20260317 21:03:13.647529 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 76 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:13.647646 30105 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 76 FOLLOWER]: Leader election lost for term 76. Reason: could not achieve majority
I20260317 21:03:13.676760 30101 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 76 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:13.676903 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 76 FOLLOWER]: Advancing to term 77
I20260317 21:03:13.677695 30101 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.677918 30101 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 77 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:13.678136 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 77 candidate_status { last_received { term: 72 index: 80 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:13.678143 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 77 candidate_status { last_received { term: 72 index: 80 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:13.678236 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 76 FOLLOWER]: Advancing to term 77
I20260317 21:03:13.678289 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 76 FOLLOWER]: Advancing to term 77
I20260317 21:03:13.678828 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 77 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 77.
I20260317 21:03:13.678874 29961 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 77 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 for term 77 because replica has last-logged OpId of term: 73 index: 80, which is greater than that of the candidate, which has last-logged OpId of term: 72 index: 80.
I20260317 21:03:13.679082 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 77 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:03:13.679236 30101 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 77 FOLLOWER]: Leader election won for term 77
I20260317 21:03:13.679313 30101 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 77 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:13.679371 30101 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 72.80, Last appended by leader: 80, Current term: 77, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.679839 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 73 to 77, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 77 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:13.695427 30107 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 77 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:13.695549 30107 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 77 FOLLOWER]: Advancing to term 78
I20260317 21:03:13.696218 30107 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.696400 30107 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 78 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:13.696630 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 78 candidate_status { last_received { term: 73 index: 80 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:13.696671 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 78 candidate_status { last_received { term: 73 index: 80 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:13.696692 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 77 FOLLOWER]: Advancing to term 78
I20260317 21:03:13.697543 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 78 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 78.
I20260317 21:03:13.697770 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 78 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:13.697888 30107 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 78 FOLLOWER]: Leader election won for term 78
I20260317 21:03:13.697944 30107 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 78 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:13.698002 30107 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 73.80, Last appended by leader: 80, Current term: 78, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.698486 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 77 to 78, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 78 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:13.777189 29887 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 77 LEADER]: Stepping down as leader of term 77
I20260317 21:03:13.777273 29887 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 77 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:13.777348 29887 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 77.81, Last appended by leader: 81, Current term: 77, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:13.777433 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 77 FOLLOWER]: Advancing to term 78
I20260317 21:03:13.778066 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 78 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 77 index: 81. Preceding OpId from leader: term: 78 index: 81. (term mismatch)
I20260317 21:03:13.778105 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 80
I20260317 21:03:13.778118 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 77.81
I20260317 21:03:13.778132 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 78 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:13.778437 30107 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 79, Time since last communication: 0.000s
I20260317 21:03:13.778502 30107 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 24015e78ad6341a9a2bc13754a55b581 log is divergent from this leader: its last log entry 72.80 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 79
I20260317 21:03:13.778826 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 79
I20260317 21:03:13.778892 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 72.80
I20260317 21:03:13.778913 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 78 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:13.779827 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 78 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 71 index: 79. Preceding OpId from leader: term: 78 index: 81. (index mismatch)
I20260317 21:03:13.780068 30107 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 79, Time since last communication: 0.000s
I20260317 21:03:13.891237 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:14.079173 30101 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 78 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:14.079345 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 78 FOLLOWER]: Advancing to term 79
I20260317 21:03:14.080134 30101 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.080344 30101 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 79 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:14.080595 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 79 candidate_status { last_received { term: 78 index: 81 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:14.080595 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 79 candidate_status { last_received { term: 78 index: 81 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:14.080899 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 79 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:14.081032 30101 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 79 FOLLOWER]: Leader election lost for term 79. Reason: could not achieve majority
I20260317 21:03:14.082410 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 79 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 78. Current term is 79. Ops: []
I20260317 21:03:14.082731 30107 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 78.81, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20260317 21:03:14.082883 30113 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 78 LEADER]: Stepping down as leader of term 78
I20260317 21:03:14.082921 30113 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 78 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:14.083006 30113 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 78.81, Last appended by leader: 81, Current term: 78, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.083096 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 78 FOLLOWER]: Advancing to term 79
I20260317 21:03:14.089924 30112 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 78 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:14.090057 30112 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 78 FOLLOWER]: Advancing to term 79
I20260317 21:03:14.090799 30112 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.090988 30112 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 79 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:14.091317 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 79 candidate_status { last_received { term: 78 index: 81 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:14.091317 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 79 candidate_status { last_received { term: 78 index: 81 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:14.091441 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 79 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 79: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:14.092008 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 79 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 79.
I20260317 21:03:14.092188 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 79 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:14.092294 30112 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 79 FOLLOWER]: Leader election won for term 79
I20260317 21:03:14.092382 30112 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 79 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:14.092435 30112 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 78.81, Last appended by leader: 81, Current term: 79, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.092978 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 78 to 79, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 79 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:14.175733 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 79 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 78 index: 81. Preceding OpId from leader: term: 79 index: 82. (index mismatch)
I20260317 21:03:14.176076 30112 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20260317 21:03:14.177201 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 79 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 78 index: 81. Preceding OpId from leader: term: 79 index: 82. (index mismatch)
I20260317 21:03:14.177444 30114 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20260317 21:03:14.382890 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 79 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:14.383111 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 79 FOLLOWER]: Advancing to term 80
I20260317 21:03:14.384052 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.384207 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 80 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 79. Current term is 80. Ops: []
I20260317 21:03:14.384339 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 80 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:14.384461 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 80 candidate_status { last_received { term: 79 index: 82 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:14.384434 30112 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 79.82, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20260317 21:03:14.384507 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 80 candidate_status { last_received { term: 79 index: 82 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:14.384608 30112 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 79 LEADER]: Stepping down as leader of term 79
I20260317 21:03:14.384655 30112 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 79 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:14.384729 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 80 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:14.384730 30112 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 79.82, Last appended by leader: 82, Current term: 79, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.384841 30112 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 79 FOLLOWER]: Advancing to term 80
I20260317 21:03:14.384891 30113 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 80 FOLLOWER]: Leader election lost for term 80. Reason: could not achieve majority
I20260317 21:03:14.447893 30101 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 79 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:14.448055 30101 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 79 FOLLOWER]: Advancing to term 80
I20260317 21:03:14.448732 30101 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.448958 30101 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 80 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:14.449261 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 80 candidate_status { last_received { term: 79 index: 82 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:14.449257 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 80 candidate_status { last_received { term: 79 index: 82 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:14.449391 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 80 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 80: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:14.449982 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 80 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 80.
I20260317 21:03:14.450225 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 80 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:14.450353 30101 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 80 FOLLOWER]: Leader election won for term 80
I20260317 21:03:14.450591 30101 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 80 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:14.450682 30101 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 79.82, Last appended by leader: 82, Current term: 80, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.451179 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 79 to 80, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 80 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:14.499914 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 80 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:14.500097 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 80 FOLLOWER]: Advancing to term 81
I20260317 21:03:14.500932 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.501176 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 81 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:14.501400 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 81 candidate_status { last_received { term: 79 index: 82 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:14.501400 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 81 candidate_status { last_received { term: 79 index: 82 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:14.501504 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 80 FOLLOWER]: Advancing to term 81
I20260317 21:03:14.502168 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 81 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 81.
I20260317 21:03:14.502444 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 81 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:14.502564 30113 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 81 FOLLOWER]: Leader election won for term 81
I20260317 21:03:14.502635 30113 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 81 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:14.502714 30113 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 79.82, Last appended by leader: 82, Current term: 81, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.503358 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 80 to 81, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 81 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:14.553773 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 81 LEADER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 80. Current term is 81. Ops: []
I20260317 21:03:14.554198 30101 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20260317 21:03:14.554339 30101 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 81.83 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 82
I20260317 21:03:14.554378 30115 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 80 LEADER]: Stepping down as leader of term 80
I20260317 21:03:14.554430 30115 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 80 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:14.554484 30115 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 80.83, Last appended by leader: 83, Current term: 80, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.554551 30115 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 80 FOLLOWER]: Advancing to term 81
I20260317 21:03:14.585783 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 81 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 80 index: 83. Preceding OpId from leader: term: 81 index: 83. (term mismatch)
I20260317 21:03:14.585893 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 82
I20260317 21:03:14.585913 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 80.83
I20260317 21:03:14.585937 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 81 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:14.586196 30113 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20260317 21:03:14.587350 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 81 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 79 index: 82. Preceding OpId from leader: term: 81 index: 83. (index mismatch)
I20260317 21:03:14.587570 30107 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20260317 21:03:14.688505 30112 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 81 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:14.688616 30112 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 81 FOLLOWER]: Advancing to term 82
I20260317 21:03:14.689316 30112 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.689541 30112 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 82 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:14.689797 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 82 candidate_status { last_received { term: 81 index: 83 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:14.689843 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 82 candidate_status { last_received { term: 81 index: 83 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:14.689913 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 81 FOLLOWER]: Advancing to term 82
I20260317 21:03:14.690459 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 82 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 82.
I20260317 21:03:14.690694 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 82 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:14.690801 30112 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 82 FOLLOWER]: Leader election won for term 82
I20260317 21:03:14.690853 30112 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 82 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:14.690902 30112 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 81.83, Last appended by leader: 83, Current term: 82, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.691423 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 81 to 82, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 82 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:14.703289 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 82 LEADER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 81. Current term is 82. Ops: []
I20260317 21:03:14.703624 30116 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 81.83, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20260317 21:03:14.703727 30116 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 82.84 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 83
I20260317 21:03:14.703750 30116 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 81 LEADER]: Stepping down as leader of term 81
I20260317 21:03:14.703765 30116 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 81 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:14.703852 30116 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 81.83, Last appended by leader: 83, Current term: 81, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.703953 30116 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 81 FOLLOWER]: Advancing to term 82
I20260317 21:03:14.782907 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 82 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 81 index: 83. Preceding OpId from leader: term: 82 index: 84. (index mismatch)
I20260317 21:03:14.783301 30112 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20260317 21:03:14.784417 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 82 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 81 index: 83. Preceding OpId from leader: term: 82 index: 84. (index mismatch)
I20260317 21:03:14.784747 30114 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20260317 21:03:14.885586 30115 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 82 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:14.885704 30115 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 82 FOLLOWER]: Advancing to term 83
I20260317 21:03:14.886447 30115 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.886646 30115 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 83 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:14.886858 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 83 candidate_status { last_received { term: 82 index: 84 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:14.886869 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 83 candidate_status { last_received { term: 82 index: 84 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:14.886957 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 82 FOLLOWER]: Advancing to term 83
I20260317 21:03:14.887594 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 83 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 83.
I20260317 21:03:14.887827 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 83 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:14.887948 30115 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 83 FOLLOWER]: Leader election won for term 83
I20260317 21:03:14.888038 30115 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 83 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:14.888103 30115 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 82.84, Last appended by leader: 84, Current term: 83, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.888698 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 82 to 83, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 83 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:14.891337 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 83 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 82. Current term is 83. Ops: []
I20260317 21:03:14.891584 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:14.891650 30114 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 82.84, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20260317 21:03:14.891755 30114 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 82 LEADER]: Stepping down as leader of term 82
I20260317 21:03:14.891774 30114 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 82 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:14.891839 30114 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 82.84, Last appended by leader: 84, Current term: 82, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:14.891917 30114 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 82 FOLLOWER]: Advancing to term 83
I20260317 21:03:14.984421 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 83 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 82 index: 84. Preceding OpId from leader: term: 83 index: 85. (index mismatch)
I20260317 21:03:14.984795 30115 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20260317 21:03:14.985918 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 83 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 82 index: 84. Preceding OpId from leader: term: 83 index: 85. (index mismatch)
I20260317 21:03:14.986109 30115 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20260317 21:03:15.179782 30114 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 83 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:15.179904 30114 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 83 FOLLOWER]: Advancing to term 84
I20260317 21:03:15.180637 30114 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.180828 30114 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 84 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:15.181221 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 84 candidate_status { last_received { term: 83 index: 85 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:15.181155 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 84 candidate_status { last_received { term: 83 index: 85 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:15.181536 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 84 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:15.181643 30114 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 84 FOLLOWER]: Leader election lost for term 84. Reason: could not achieve majority
I20260317 21:03:15.199995 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 84 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 83. Current term is 84. Ops: []
I20260317 21:03:15.200302 30101 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 83.85, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20260317 21:03:15.200452 30118 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 83 LEADER]: Stepping down as leader of term 83
I20260317 21:03:15.200528 30118 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 83 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:15.200601 30118 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 83.85, Last appended by leader: 85, Current term: 83, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.200670 30118 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 83 FOLLOWER]: Advancing to term 84
I20260317 21:03:15.249573 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 83 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:15.249687 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 83 FOLLOWER]: Advancing to term 84
I20260317 21:03:15.250473 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.250706 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 84 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:15.250950 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 84 candidate_status { last_received { term: 83 index: 85 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:15.250986 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 84 candidate_status { last_received { term: 83 index: 85 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:15.251096 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 84 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 84: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:15.251636 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 84 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 84.
I20260317 21:03:15.251881 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 84 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:15.251999 30113 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 84 FOLLOWER]: Leader election won for term 84
I20260317 21:03:15.252056 30113 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 84 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:15.252090 30113 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 83.85, Last appended by leader: 85, Current term: 84, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.252635 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 83 to 84, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 84 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:15.315219 30114 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 84 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:15.315343 30114 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 84 FOLLOWER]: Advancing to term 85
I20260317 21:03:15.316104 30114 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.316334 30114 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 85 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:15.316645 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 85 candidate_status { last_received { term: 83 index: 85 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:15.316730 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 85 candidate_status { last_received { term: 83 index: 85 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:15.316856 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 84 FOLLOWER]: Advancing to term 85
I20260317 21:03:15.317546 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 85 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 85.
I20260317 21:03:15.317751 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 85 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:15.317857 30114 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 85 FOLLOWER]: Leader election won for term 85
I20260317 21:03:15.317950 30114 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 85 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:15.318006 30114 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 83.85, Last appended by leader: 85, Current term: 85, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.318512 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 84 to 85, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 85 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:15.352237 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 85 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 84. Current term is 85. Ops: []
I20260317 21:03:15.352526 30113 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20260317 21:03:15.352638 30113 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 84 LEADER]: Stepping down as leader of term 84
I20260317 21:03:15.352658 30113 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 84 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:15.352730 30113 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 84.86, Last appended by leader: 86, Current term: 84, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.352810 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 84 FOLLOWER]: Advancing to term 85
I20260317 21:03:15.407339 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 85 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 83 index: 85. Preceding OpId from leader: term: 85 index: 86. (index mismatch)
I20260317 21:03:15.407624 30114 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20260317 21:03:15.408857 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 85 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 84 index: 86. Preceding OpId from leader: term: 85 index: 86. (term mismatch)
I20260317 21:03:15.408914 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 85
I20260317 21:03:15.408939 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 84.86
I20260317 21:03:15.408959 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 85 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:15.409119 30119 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20260317 21:03:15.668663 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 85 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:15.668787 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 85 FOLLOWER]: Advancing to term 86
I20260317 21:03:15.670337 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.670600 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 86 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:15.670768 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 86 candidate_status { last_received { term: 85 index: 86 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:15.670768 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 86 candidate_status { last_received { term: 85 index: 86 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:15.671021 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 86 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:15.671124 30113 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 86 FOLLOWER]: Leader election lost for term 86. Reason: could not achieve majority
I20260317 21:03:15.674104 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 86 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 85. Current term is 86. Ops: []
I20260317 21:03:15.674443 30114 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 85.86, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20260317 21:03:15.674580 30112 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 85 LEADER]: Stepping down as leader of term 85
I20260317 21:03:15.674628 30112 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 85 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:15.674693 30112 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 85.86, Last appended by leader: 86, Current term: 85, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.674750 30112 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 85 FOLLOWER]: Advancing to term 86
I20260317 21:03:15.695282 30118 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 85 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:15.695418 30118 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 85 FOLLOWER]: Advancing to term 86
I20260317 21:03:15.696308 30118 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.696532 30118 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 86 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:15.696766 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 86 candidate_status { last_received { term: 85 index: 86 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:15.696766 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 86 candidate_status { last_received { term: 85 index: 86 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:15.697038 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 86 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 86: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:15.697670 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 86 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 86.
I20260317 21:03:15.698130 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 86 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:15.698241 30118 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 86 FOLLOWER]: Leader election won for term 86
I20260317 21:03:15.698302 30118 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 86 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:15.698338 30118 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 85.86, Last appended by leader: 86, Current term: 86, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.698812 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 85 to 86, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 86 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:15.775573 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 86 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:15.775705 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 86 FOLLOWER]: Advancing to term 87
I20260317 21:03:15.776451 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.776678 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 87 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:15.776907 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 87 candidate_status { last_received { term: 85 index: 86 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:15.776925 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 87 candidate_status { last_received { term: 85 index: 86 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:15.776991 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 86 FOLLOWER]: Advancing to term 87
I20260317 21:03:15.777688 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 87 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 87.
I20260317 21:03:15.777910 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 87 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:15.778017 30113 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 87 FOLLOWER]: Leader election won for term 87
I20260317 21:03:15.778069 30113 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 87 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:15.778122 30113 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 85.86, Last appended by leader: 86, Current term: 87, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.778769 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 86 to 87, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 87 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:15.787405 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 87 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 86. Current term is 87. Ops: []
I20260317 21:03:15.787721 30118 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20260317 21:03:15.787843 30118 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 86 LEADER]: Stepping down as leader of term 86
I20260317 21:03:15.787861 30118 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 86 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:15.787931 30118 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 86.87, Last appended by leader: 87, Current term: 86, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.788007 30118 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 86 FOLLOWER]: Advancing to term 87
I20260317 21:03:15.856256 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 87 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 86 index: 87. Preceding OpId from leader: term: 87 index: 87. (term mismatch)
I20260317 21:03:15.856349 29887 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 86
I20260317 21:03:15.856371 29887 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 86.87
I20260317 21:03:15.856393 29887 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 87 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:15.856654 30113 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20260317 21:03:15.857906 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 87 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 85 index: 86. Preceding OpId from leader: term: 87 index: 87. (index mismatch)
I20260317 21:03:15.858144 30120 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20260317 21:03:15.891851 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:15.958882 30112 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 87 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:15.958992 30112 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 87 FOLLOWER]: Advancing to term 88
I20260317 21:03:15.959687 30112 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.959858 30112 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 88 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:15.960131 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 88 candidate_status { last_received { term: 87 index: 87 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:15.960167 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 88 candidate_status { last_received { term: 87 index: 87 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:15.960292 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 87 FOLLOWER]: Advancing to term 88
I20260317 21:03:15.960888 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 88 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 88.
I20260317 21:03:15.961112 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 88 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:15.961252 30112 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 88 FOLLOWER]: Leader election won for term 88
I20260317 21:03:15.961345 30112 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 88 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:15.961406 30112 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 87.87, Last appended by leader: 87, Current term: 88, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.961956 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 87 to 88, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 88 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:15.968786 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 88 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 87. Current term is 88. Ops: []
I20260317 21:03:15.969267 30120 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 87.87, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20260317 21:03:15.969430 30113 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 87 LEADER]: Stepping down as leader of term 87
I20260317 21:03:15.969483 30113 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 87 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:15.969554 30113 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 87.87, Last appended by leader: 87, Current term: 87, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:15.969640 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 87 FOLLOWER]: Advancing to term 88
I20260317 21:03:16.060663 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 88 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 87 index: 87. Preceding OpId from leader: term: 88 index: 88. (index mismatch)
I20260317 21:03:16.061033 30114 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20260317 21:03:16.061280 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 88 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 87 index: 87. Preceding OpId from leader: term: 88 index: 88. (index mismatch)
I20260317 21:03:16.061497 30114 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20260317 21:03:16.380060 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 88 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:16.380172 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 88 FOLLOWER]: Advancing to term 89
I20260317 21:03:16.380935 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:16.381158 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 89 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:16.381434 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 89 candidate_status { last_received { term: 88 index: 88 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:16.381443 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 89 candidate_status { last_received { term: 88 index: 88 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:16.381702 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 89 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:16.381798 30113 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 89 FOLLOWER]: Leader election lost for term 89. Reason: could not achieve majority
I20260317 21:03:16.404011 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 89 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 88. Current term is 89. Ops: []
I20260317 21:03:16.404436 30114 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 88.88, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20260317 21:03:16.404613 30114 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 88 LEADER]: Stepping down as leader of term 88
I20260317 21:03:16.404649 30114 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 88 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:16.404714 30114 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 88.88, Last appended by leader: 88, Current term: 88, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:16.404793 30114 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 88 FOLLOWER]: Advancing to term 89
I20260317 21:03:16.405915 30118 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 88 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:16.405973 30118 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 88 FOLLOWER]: Advancing to term 89
I20260317 21:03:16.406572 30118 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:16.406733 30118 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 89 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:16.406888 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 89 candidate_status { last_received { term: 88 index: 88 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:16.406968 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 89 candidate_status { last_received { term: 88 index: 88 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:16.407086 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 89 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 89: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:16.407598 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 89 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 89.
I20260317 21:03:16.407821 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 89 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:16.407961 30118 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 89 FOLLOWER]: Leader election won for term 89
I20260317 21:03:16.408069 30118 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 89 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:16.408165 30118 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 88.88, Last appended by leader: 88, Current term: 89, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:16.408857 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 88 to 89, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 89 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:16.496448 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 89 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 88 index: 88. Preceding OpId from leader: term: 89 index: 89. (index mismatch)
I20260317 21:03:16.496819 30118 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20260317 21:03:16.498008 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 89 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 88 index: 88. Preceding OpId from leader: term: 89 index: 89. (index mismatch)
I20260317 21:03:16.498248 30118 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20260317 21:03:16.599545 30123 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 89 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:16.599647 30123 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 89 FOLLOWER]: Advancing to term 90
I20260317 21:03:16.600425 30123 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:16.600629 30123 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 90 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:16.600904 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 90 candidate_status { last_received { term: 89 index: 89 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:16.601001 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 89 FOLLOWER]: Advancing to term 90
I20260317 21:03:16.600912 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 90 candidate_status { last_received { term: 89 index: 89 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:16.601712 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 90 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 90.
I20260317 21:03:16.601979 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 90 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:16.602128 30123 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 90 FOLLOWER]: Leader election won for term 90
I20260317 21:03:16.602226 30123 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 90 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:16.602301 30123 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 89.89, Last appended by leader: 89, Current term: 90, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:16.602890 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 89 to 90, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 90 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:16.616317 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 90 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 89. Current term is 90. Ops: []
I20260317 21:03:16.616717 30118 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 89.89, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20260317 21:03:16.616894 30122 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 89 LEADER]: Stepping down as leader of term 89
I20260317 21:03:16.616952 30122 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 89 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:16.617033 30122 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 89.89, Last appended by leader: 89, Current term: 89, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:16.617121 30122 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 89 FOLLOWER]: Advancing to term 90
I20260317 21:03:16.679479 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 90 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 89 index: 89. Preceding OpId from leader: term: 90 index: 90. (index mismatch)
I20260317 21:03:16.679888 30123 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20260317 21:03:16.681094 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 90 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 89 index: 89. Preceding OpId from leader: term: 90 index: 90. (index mismatch)
I20260317 21:03:16.681365 30113 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20260317 21:03:16.781981 30114 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 90 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:16.782109 30114 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 90 FOLLOWER]: Advancing to term 91
I20260317 21:03:16.782855 30114 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:16.783031 30114 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 91 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:16.783351 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 91 candidate_status { last_received { term: 90 index: 90 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:16.783461 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 91 candidate_status { last_received { term: 90 index: 90 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:16.783587 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 90 FOLLOWER]: Advancing to term 91
I20260317 21:03:16.784137 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 91 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 91.
I20260317 21:03:16.784392 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 91 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:16.784524 30114 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 91 FOLLOWER]: Leader election won for term 91
I20260317 21:03:16.784617 30114 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 91 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:16.784683 30114 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 90.90, Last appended by leader: 90, Current term: 91, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:16.785143 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 90 to 91, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 91 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:16.787829 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 91 LEADER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 90. Current term is 91. Ops: []
I20260317 21:03:16.788095 30113 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 90.90, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20260317 21:03:16.788249 30113 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 91.91 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 90
I20260317 21:03:16.788293 30123 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 90 LEADER]: Stepping down as leader of term 90
I20260317 21:03:16.788338 30123 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 90 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:16.788403 30123 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 90.90, Last appended by leader: 90, Current term: 90, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:16.788492 30123 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 90 FOLLOWER]: Advancing to term 91
I20260317 21:03:16.892148 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:16.901083 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 91 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 90 index: 90. Preceding OpId from leader: term: 91 index: 91. (index mismatch)
I20260317 21:03:16.901506 30114 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20260317 21:03:16.902694 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 91 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 90 index: 90. Preceding OpId from leader: term: 91 index: 91. (index mismatch)
I20260317 21:03:16.902910 30124 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20260317 21:03:17.004134 30122 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 91 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:17.004257 30122 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 91 FOLLOWER]: Advancing to term 92
I20260317 21:03:17.005129 30122 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.005396 30122 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 92 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:17.005601 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 92 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:17.005604 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 92 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:17.005895 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 92 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:17.006000 30122 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 92 FOLLOWER]: Leader election lost for term 92. Reason: could not achieve majority
I20260317 21:03:17.019819 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 92 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 91. Current term is 92. Ops: []
I20260317 21:03:17.020128 30114 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 91.91, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20260317 21:03:17.020275 30114 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 91 LEADER]: Stepping down as leader of term 91
I20260317 21:03:17.020304 30114 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 91 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:17.020394 30114 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 91.91, Last appended by leader: 91, Current term: 91, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.020478 30114 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 91 FOLLOWER]: Advancing to term 92
I20260317 21:03:17.101357 30123 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 91 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:17.101475 30123 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 91 FOLLOWER]: Advancing to term 92
I20260317 21:03:17.102275 30123 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.102456 30123 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 92 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:17.102682 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 92 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:17.102707 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 92 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:17.102859 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 92 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 92: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:17.103381 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 92 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 92.
I20260317 21:03:17.103612 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 92 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:17.103708 30123 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 92 FOLLOWER]: Leader election won for term 92
I20260317 21:03:17.103780 30123 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 92 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:17.103837 30123 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 91.91, Last appended by leader: 91, Current term: 92, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.104347 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 91 to 92, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 92 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:17.120208 30122 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 92 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:17.120321 30122 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 92 FOLLOWER]: Advancing to term 93
I20260317 21:03:17.121018 30122 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.121243 30122 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 93 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:17.121451 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 93 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:17.121485 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 93 candidate_status { last_received { term: 91 index: 91 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:17.121605 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 92 FOLLOWER]: Advancing to term 93
I20260317 21:03:17.122195 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 93 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 93.
I20260317 21:03:17.122407 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 93 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:17.122520 30122 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 93 FOLLOWER]: Leader election won for term 93
I20260317 21:03:17.122606 30122 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 93 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:17.122663 30122 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 91.91, Last appended by leader: 91, Current term: 93, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.123152 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 92 to 93, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 93 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:17.212688 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 93 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 91 index: 91. Preceding OpId from leader: term: 93 index: 92. (index mismatch)
I20260317 21:03:17.212990 30122 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20260317 21:03:17.213594 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 93 LEADER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 92. Current term is 93. Ops: []
I20260317 21:03:17.213843 30123 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20260317 21:03:17.213925 30123 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 24015e78ad6341a9a2bc13754a55b581 log is divergent from this leader: its last log entry 93.92 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 91
I20260317 21:03:17.213964 30123 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 92 LEADER]: Stepping down as leader of term 92
I20260317 21:03:17.213989 30123 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 92 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:17.214030 30123 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 92.92, Last appended by leader: 92, Current term: 92, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.214107 30123 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 92 FOLLOWER]: Advancing to term 93
I20260317 21:03:17.215101 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 93 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 92 index: 92. Preceding OpId from leader: term: 93 index: 92. (term mismatch)
I20260317 21:03:17.215145 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 91
I20260317 21:03:17.215159 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 92.92
I20260317 21:03:17.215173 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 93 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:17.215627 30118 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20260317 21:03:17.315316 30114 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 93 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:17.315425 30114 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 93 FOLLOWER]: Advancing to term 94
I20260317 21:03:17.316124 30114 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.316278 30114 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 94 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:17.316567 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:17.316601 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:17.316833 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 94 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:17.316926 30114 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 94 FOLLOWER]: Leader election lost for term 94. Reason: could not achieve majority
I20260317 21:03:17.326431 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 94 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 93. Current term is 94. Ops: []
I20260317 21:03:17.326726 30118 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 93.92, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20260317 21:03:17.326879 30118 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 93 LEADER]: Stepping down as leader of term 93
I20260317 21:03:17.326916 30118 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 93 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:17.326988 30118 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 93.92, Last appended by leader: 92, Current term: 93, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.327075 30118 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 93 FOLLOWER]: Advancing to term 94
I20260317 21:03:17.439357 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 93 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:17.439474 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 93 FOLLOWER]: Advancing to term 94
I20260317 21:03:17.440161 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.440349 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 94 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:17.440603 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:17.440702 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 94 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 94: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:17.440611 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:17.441493 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 94 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 94.
I20260317 21:03:17.441684 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 94 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:17.441800 30113 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 94 FOLLOWER]: Leader election won for term 94
I20260317 21:03:17.441898 30113 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 94 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:17.441958 30113 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 93.92, Last appended by leader: 92, Current term: 94, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.442420 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 93 to 94, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 94 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:17.444316 30114 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 94 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:17.444419 30114 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 94 FOLLOWER]: Advancing to term 95
I20260317 21:03:17.444998 30114 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.445140 30114 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 95 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:17.445362 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 95 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:17.445366 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 95 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:17.445479 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 94 FOLLOWER]: Advancing to term 95
I20260317 21:03:17.446014 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 95.
I20260317 21:03:17.446254 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 95 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:17.446362 30114 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 95 FOLLOWER]: Leader election won for term 95
I20260317 21:03:17.446442 30114 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 95 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:17.446512 30114 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 93.92, Last appended by leader: 92, Current term: 95, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.447185 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 94 to 95, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 95 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:17.529363 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 95 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 94. Current term is 95. Ops: []
I20260317 21:03:17.529808 30113 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20260317 21:03:17.529923 30113 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 94 LEADER]: Stepping down as leader of term 94
I20260317 21:03:17.529942 30113 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 94 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:17.530016 30113 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 94.93, Last appended by leader: 93, Current term: 94, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.530090 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 94 FOLLOWER]: Advancing to term 95
I20260317 21:03:17.532286 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 95 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 93 index: 92. Preceding OpId from leader: term: 95 index: 93. (index mismatch)
I20260317 21:03:17.532541 30114 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20260317 21:03:17.533615 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 95 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 94 index: 93. Preceding OpId from leader: term: 95 index: 93. (term mismatch)
I20260317 21:03:17.533670 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 92
I20260317 21:03:17.533699 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 94.93
I20260317 21:03:17.533718 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 95 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:17.533860 30114 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20260317 21:03:17.722240 30118 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 95 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:17.722371 30118 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 95 FOLLOWER]: Advancing to term 96
I20260317 21:03:17.723096 30118 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.723289 30118 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 96 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:17.723464 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 96 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:17.723503 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 96 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:17.723773 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 96 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:17.723867 30118 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 96 FOLLOWER]: Leader election lost for term 96. Reason: could not achieve majority
I20260317 21:03:17.731076 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 96 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 95. Current term is 96. Ops: []
I20260317 21:03:17.731448 30125 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 95.93, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20260317 21:03:17.731570 30125 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 95 LEADER]: Stepping down as leader of term 95
I20260317 21:03:17.731590 30125 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 95 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:17.731681 30125 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 95.93, Last appended by leader: 93, Current term: 95, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.731770 30125 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 95 FOLLOWER]: Advancing to term 96
I20260317 21:03:17.755666 30123 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 95 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:17.755972 30123 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 95 FOLLOWER]: Advancing to term 96
I20260317 21:03:17.756851 30123 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.757077 30123 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 96 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:17.757404 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 96 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:17.757404 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 96 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:17.757591 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 96 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 96: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:17.758147 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 96 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 96.
I20260317 21:03:17.758374 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 96 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:17.758489 30123 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 96 FOLLOWER]: Leader election won for term 96
I20260317 21:03:17.758566 30123 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 96 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:17.758610 30123 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 95.93, Last appended by leader: 93, Current term: 96, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.759203 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 95 to 96, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 96 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:17.848839 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 96 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 95 index: 93. Preceding OpId from leader: term: 96 index: 94. (index mismatch)
I20260317 21:03:17.849279 30123 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20260317 21:03:17.849293 30118 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 96 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:17.849404 30118 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 96 FOLLOWER]: Advancing to term 97
I20260317 21:03:17.850142 30118 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.850325 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 97 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 96. Current term is 97. Ops: [96.94-96.94]
I20260317 21:03:17.850324 30118 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 97 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:17.850452 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 97 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:17.850476 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 97 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:17.850523 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 96 FOLLOWER]: Advancing to term 97
I20260317 21:03:17.850595 30123 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 95.93, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20260317 21:03:17.850716 30113 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 96 LEADER]: Stepping down as leader of term 96
I20260317 21:03:17.850762 30113 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 96 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:17.850812 30113 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 96.94, Last appended by leader: 94, Current term: 96, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.850878 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 96 FOLLOWER]: Advancing to term 97
I20260317 21:03:17.851301 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 97 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 97.
I20260317 21:03:17.851555 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 97 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:17.851660 30118 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 97 FOLLOWER]: Leader election won for term 97
I20260317 21:03:17.851761 30118 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 97 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:17.851840 30118 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 95.93, Last appended by leader: 93, Current term: 97, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:17.852398 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 96 to 97, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 97 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:17.892508 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:17.950941 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 97 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 96 index: 94. Preceding OpId from leader: term: 97 index: 94. (term mismatch)
I20260317 21:03:17.951021 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 93
I20260317 21:03:17.951035 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 96.94
I20260317 21:03:17.951050 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 97 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:17.951323 30118 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20260317 21:03:17.952486 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 97 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 95 index: 93. Preceding OpId from leader: term: 97 index: 94. (index mismatch)
I20260317 21:03:17.952742 30127 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20260317 21:03:18.148909 30125 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 97 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:18.149030 30125 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 97 FOLLOWER]: Advancing to term 98
I20260317 21:03:18.149853 30125 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.149992 30125 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 98 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:18.150313 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 98 candidate_status { last_received { term: 97 index: 94 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:18.150313 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 98 candidate_status { last_received { term: 97 index: 94 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:18.150616 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 98 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:18.150723 30125 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 98 FOLLOWER]: Leader election lost for term 98. Reason: could not achieve majority
I20260317 21:03:18.170199 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 98 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 97. Current term is 98. Ops: []
I20260317 21:03:18.170490 30127 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 97.94, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20260317 21:03:18.170655 30126 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 97 LEADER]: Stepping down as leader of term 97
I20260317 21:03:18.170703 30126 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 97 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:18.170826 30126 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 97.94, Last appended by leader: 94, Current term: 97, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.170912 30126 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 97 FOLLOWER]: Advancing to term 98
I20260317 21:03:18.264313 30125 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 98 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:18.264423 30125 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 98 FOLLOWER]: Advancing to term 99
I20260317 21:03:18.265234 30125 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.265492 30125 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 99 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:18.265739 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 99 candidate_status { last_received { term: 97 index: 94 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:18.265769 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 99 candidate_status { last_received { term: 97 index: 94 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:18.265906 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 98 FOLLOWER]: Advancing to term 99
I20260317 21:03:18.266506 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 99 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 99.
I20260317 21:03:18.266760 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 99 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:18.266911 30125 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 99 FOLLOWER]: Leader election won for term 99
I20260317 21:03:18.267071 30125 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 99 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:18.267141 30125 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 97.94, Last appended by leader: 94, Current term: 99, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.267664 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 97 to 99, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 99 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:18.268893 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 97 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:18.268942 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 97 FOLLOWER]: Advancing to term 98
I20260317 21:03:18.269594 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.269886 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 98 candidate_status { last_received { term: 97 index: 94 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:18.269891 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 98 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:18.269946 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 98 candidate_status { last_received { term: 97 index: 94 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:18.270116 29887 raft_consensus.cc:2368] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 98. Current term is 99.
I20260317 21:03:18.270308 29926 leader_election.cc:400] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 98 election: Vote denied by peer 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653) with higher term. Message: Invalid argument: T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 98. Current term is 99.
I20260317 21:03:18.270354 29926 leader_election.cc:403] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 98 election: Cancelling election due to peer responding with higher term
I20260317 21:03:18.270412 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 98 FOLLOWER]: Advancing to term 99
I20260317 21:03:18.270999 30113 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 99 FOLLOWER]: Leader election lost for term 98. Reason: Vote denied by peer 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653) with higher term. Message: Invalid argument: T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 98. Current term is 99.
I20260317 21:03:18.348194 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 99 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 97 index: 94. Preceding OpId from leader: term: 99 index: 95. (index mismatch)
I20260317 21:03:18.348868 30125 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20260317 21:03:18.350517 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 99 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 97 index: 94. Preceding OpId from leader: term: 99 index: 95. (index mismatch)
I20260317 21:03:18.350733 30129 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20260317 21:03:18.536198 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 99 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:18.536373 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 99 FOLLOWER]: Advancing to term 100
I20260317 21:03:18.537043 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.537304 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 100 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:18.537498 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 100 candidate_status { last_received { term: 99 index: 95 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:18.537499 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 100 candidate_status { last_received { term: 99 index: 95 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:18.537784 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 100 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:18.537938 30113 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 100 FOLLOWER]: Leader election lost for term 100. Reason: could not achieve majority
I20260317 21:03:18.542958 30126 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 99 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:18.543051 30126 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 99 FOLLOWER]: Advancing to term 100
I20260317 21:03:18.543782 30126 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.543954 30126 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 100 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:18.544143 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 100 candidate_status { last_received { term: 99 index: 95 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:18.544162 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 100 candidate_status { last_received { term: 99 index: 95 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:18.544262 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 100 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 100: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:18.544483 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 100 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:18.544600 30126 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 100 FOLLOWER]: Leader election lost for term 100. Reason: could not achieve majority
I20260317 21:03:18.550690 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 100 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 99. Current term is 100. Ops: []
I20260317 21:03:18.550976 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 99.95, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20260317 21:03:18.551090 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 99 LEADER]: Stepping down as leader of term 99
I20260317 21:03:18.551110 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 99 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:18.551194 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 99.95, Last appended by leader: 95, Current term: 99, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.551298 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 99 FOLLOWER]: Advancing to term 100
I20260317 21:03:18.663103 30126 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 100 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:18.663208 30126 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 100 FOLLOWER]: Advancing to term 101
I20260317 21:03:18.663954 30126 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.664139 30126 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 101 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:18.664430 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 101 candidate_status { last_received { term: 99 index: 95 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:18.664358 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 101 candidate_status { last_received { term: 99 index: 95 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:18.664506 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 100 FOLLOWER]: Advancing to term 101
I20260317 21:03:18.664513 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 100 FOLLOWER]: Advancing to term 101
I20260317 21:03:18.665277 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 101 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 101.
I20260317 21:03:18.665277 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 101 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 101.
I20260317 21:03:18.665573 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 101 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters:
I20260317 21:03:18.665701 30126 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 101 FOLLOWER]: Leader election won for term 101
I20260317 21:03:18.665766 30126 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 101 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:18.665802 30126 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 99.95, Last appended by leader: 95, Current term: 101, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.666203 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 99 to 101, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 101 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:18.757440 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 101 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 99 index: 95. Preceding OpId from leader: term: 101 index: 96. (index mismatch)
I20260317 21:03:18.757853 30127 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20260317 21:03:18.759083 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 101 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 99 index: 95. Preceding OpId from leader: term: 101 index: 96. (index mismatch)
I20260317 21:03:18.759369 30130 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20260317 21:03:18.859987 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 101 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:18.860105 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 101 FOLLOWER]: Advancing to term 102
I20260317 21:03:18.860293 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 101 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:18.860384 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 101 FOLLOWER]: Advancing to term 102
I20260317 21:03:18.861030 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.861425 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.861482 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 102 candidate_status { last_received { term: 101 index: 96 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:18.861483 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 102 candidate_status { last_received { term: 101 index: 96 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:18.861600 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 102 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:18.861825 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 102 candidate_status { last_received { term: 101 index: 96 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:18.861850 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 102 candidate_status { last_received { term: 101 index: 96 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:18.861919 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 102 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 102: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:18.861613 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 102 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:18.862039 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 102 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:18.861601 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 102 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 102: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:18.862257 30128 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 102 FOLLOWER]: Leader election lost for term 102. Reason: could not achieve majority
I20260317 21:03:18.862354 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 102 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:18.862473 30113 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 102 FOLLOWER]: Leader election lost for term 102. Reason: could not achieve majority
I20260317 21:03:18.873231 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 102 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 101. Current term is 102. Ops: []
I20260317 21:03:18.873602 30127 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 101.96, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20260317 21:03:18.873775 30127 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 101 LEADER]: Stepping down as leader of term 101
I20260317 21:03:18.873818 30127 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 101 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:18.873903 30127 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 101.96, Last appended by leader: 96, Current term: 101, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:18.873989 30127 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 101 FOLLOWER]: Advancing to term 102
I20260317 21:03:18.893146 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:18.999900 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 102 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:19.000006 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 102 FOLLOWER]: Advancing to term 103
I20260317 21:03:19.000779 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.001097 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 103 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:19.001238 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 103 candidate_status { last_received { term: 101 index: 96 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:19.001237 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 103 candidate_status { last_received { term: 101 index: 96 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:19.001358 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 102 FOLLOWER]: Advancing to term 103
I20260317 21:03:19.001389 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 102 FOLLOWER]: Advancing to term 103
I20260317 21:03:19.002245 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 103 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 103.
I20260317 21:03:19.002244 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 103 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 103.
I20260317 21:03:19.002492 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 103 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:03:19.002637 30113 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 103 FOLLOWER]: Leader election won for term 103
I20260317 21:03:19.002743 30113 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 103 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:19.002821 30113 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 101.96, Last appended by leader: 96, Current term: 103, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.003291 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 101 to 103, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 103 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:19.103684 30127 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 103 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:19.103794 30127 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 103 FOLLOWER]: Advancing to term 104
I20260317 21:03:19.104635 30127 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.104867 30127 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 104 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:19.105062 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 104 candidate_status { last_received { term: 101 index: 96 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:19.105108 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 104 candidate_status { last_received { term: 101 index: 96 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:19.105209 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 103 FOLLOWER]: Advancing to term 104
I20260317 21:03:19.105852 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 104 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 104.
I20260317 21:03:19.106117 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 104 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:19.106228 30127 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 104 FOLLOWER]: Leader election won for term 104
I20260317 21:03:19.106290 30127 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 104 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:19.106343 30127 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 101.96, Last appended by leader: 96, Current term: 104, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.106875 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 103 to 104, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 104 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:19.109797 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 104 LEADER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 103. Current term is 104. Ops: []
I20260317 21:03:19.110060 30113 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20260317 21:03:19.110175 30113 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 24015e78ad6341a9a2bc13754a55b581 log is divergent from this leader: its last log entry 104.97 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 96
I20260317 21:03:19.110211 30113 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 103 LEADER]: Stepping down as leader of term 103
I20260317 21:03:19.110260 30113 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 103 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:19.110325 30113 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 103.97, Last appended by leader: 97, Current term: 103, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.110415 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 103 FOLLOWER]: Advancing to term 104
I20260317 21:03:19.218487 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 104 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 103 index: 97. Preceding OpId from leader: term: 104 index: 97. (term mismatch)
I20260317 21:03:19.218577 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 96
I20260317 21:03:19.218591 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 103.97
I20260317 21:03:19.218606 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 104 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:19.218888 30127 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20260317 21:03:19.219981 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 104 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 101 index: 96. Preceding OpId from leader: term: 104 index: 97. (index mismatch)
I20260317 21:03:19.220157 30127 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20260317 21:03:19.335687 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 104 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:19.335808 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 104 FOLLOWER]: Advancing to term 105
I20260317 21:03:19.336512 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.336715 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 105 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:19.336917 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 105 candidate_status { last_received { term: 104 index: 97 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:19.336959 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 105 candidate_status { last_received { term: 104 index: 97 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:19.337327 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 105 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:19.337482 30113 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 105 FOLLOWER]: Leader election lost for term 105. Reason: could not achieve majority
I20260317 21:03:19.343039 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 105 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 104. Current term is 105. Ops: []
I20260317 21:03:19.343416 30127 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 104.97, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20260317 21:03:19.343592 30126 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 104 LEADER]: Stepping down as leader of term 104
I20260317 21:03:19.343632 30126 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 104 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:19.343709 30126 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 104.97, Last appended by leader: 97, Current term: 104, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.343798 30126 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 104 FOLLOWER]: Advancing to term 105
I20260317 21:03:19.422835 30132 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 104 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:19.422950 30132 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 104 FOLLOWER]: Advancing to term 105
I20260317 21:03:19.423707 30132 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.423846 30132 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 105 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:19.424156 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 105 candidate_status { last_received { term: 104 index: 97 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:19.424237 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 105 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 105: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:19.424161 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 105 candidate_status { last_received { term: 104 index: 97 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:19.424834 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 105 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 105.
I20260317 21:03:19.425052 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 105 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:19.425156 30132 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 105 FOLLOWER]: Leader election won for term 105
I20260317 21:03:19.425307 30132 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 105 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:19.425369 30132 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 104.97, Last appended by leader: 97, Current term: 105, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.425891 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 104 to 105, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 105 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:19.456022 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 105 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:19.456130 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 105 FOLLOWER]: Advancing to term 106
I20260317 21:03:19.456837 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 106 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.457033 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 106 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:19.457280 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 106 candidate_status { last_received { term: 104 index: 97 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:19.457279 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 106 candidate_status { last_received { term: 104 index: 97 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:19.457418 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 105 FOLLOWER]: Advancing to term 106
I20260317 21:03:19.457958 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 106 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 106.
I20260317 21:03:19.458207 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 106 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:19.458360 30113 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 106 FOLLOWER]: Leader election won for term 106
I20260317 21:03:19.458443 30113 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 106 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:19.458482 30113 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 104.97, Last appended by leader: 97, Current term: 106, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.459076 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 105 to 106, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 106 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:19.533310 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 106 LEADER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 105. Current term is 106. Ops: []
I20260317 21:03:19.533638 30132 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20260317 21:03:19.533758 30132 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 106.98 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 97
I20260317 21:03:19.533787 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 105 LEADER]: Stepping down as leader of term 105
I20260317 21:03:19.533826 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 105 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:19.533890 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 105.98, Last appended by leader: 98, Current term: 105, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.533960 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 105 FOLLOWER]: Advancing to term 106
I20260317 21:03:19.573544 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 106 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 104 index: 97. Preceding OpId from leader: term: 106 index: 98. (index mismatch)
I20260317 21:03:19.573849 30113 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20260317 21:03:19.575001 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 106 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 105 index: 98. Preceding OpId from leader: term: 106 index: 98. (term mismatch)
I20260317 21:03:19.575058 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 97
I20260317 21:03:19.575078 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 105.98
I20260317 21:03:19.575098 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 106 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:19.575300 30133 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20260317 21:03:19.675598 30126 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 106 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:19.675720 30126 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 106 FOLLOWER]: Advancing to term 107
I20260317 21:03:19.676558 30126 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.676772 30126 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 107 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:19.677019 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 107 candidate_status { last_received { term: 106 index: 98 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:19.677019 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 107 candidate_status { last_received { term: 106 index: 98 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:19.677372 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 107 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:19.677484 30126 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 107 FOLLOWER]: Leader election lost for term 107. Reason: could not achieve majority
I20260317 21:03:19.694121 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 107 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 106. Current term is 107. Ops: []
I20260317 21:03:19.694515 30113 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 106.98, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20260317 21:03:19.694638 30113 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 106 LEADER]: Stepping down as leader of term 106
I20260317 21:03:19.694657 30113 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 106 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:19.694732 30113 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 106.98, Last appended by leader: 98, Current term: 106, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.694821 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 106 FOLLOWER]: Advancing to term 107
I20260317 21:03:19.759727 30132 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 106 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:19.759837 30132 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 106 FOLLOWER]: Advancing to term 107
I20260317 21:03:19.760593 30132 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.760771 30132 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 107 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:19.761072 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 107 candidate_status { last_received { term: 106 index: 98 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:19.761030 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 107 candidate_status { last_received { term: 106 index: 98 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:19.761269 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 107 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 107: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:19.761852 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 107 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 107.
I20260317 21:03:19.762064 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 107 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:19.762169 30132 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 107 FOLLOWER]: Leader election won for term 107
I20260317 21:03:19.762256 30132 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 107 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:19.762317 30132 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 106.98, Last appended by leader: 98, Current term: 107, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.762825 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 106 to 107, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 107 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:19.827855 30126 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 107 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:19.827975 30126 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 107 FOLLOWER]: Advancing to term 108
I20260317 21:03:19.828644 30126 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.828850 30126 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 108 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:19.829090 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 108 candidate_status { last_received { term: 106 index: 98 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:19.829103 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 108 candidate_status { last_received { term: 106 index: 98 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:19.829154 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 107 FOLLOWER]: Advancing to term 108
I20260317 21:03:19.829859 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 108 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 108.
I20260317 21:03:19.830058 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 108 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:19.830189 30126 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 108 FOLLOWER]: Leader election won for term 108
I20260317 21:03:19.830271 30126 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 108 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:19.830325 30126 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 106.98, Last appended by leader: 98, Current term: 108, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.830808 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 107 to 108, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 108 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:19.866384 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 108 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 107. Current term is 108. Ops: []
I20260317 21:03:19.866791 30132 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20260317 21:03:19.866940 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 107 LEADER]: Stepping down as leader of term 107
I20260317 21:03:19.866987 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 107 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:19.867041 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 107.99, Last appended by leader: 99, Current term: 107, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:19.867111 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 107 FOLLOWER]: Advancing to term 108
I20260317 21:03:19.893492 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:19.933539 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 108 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 107 index: 99. Preceding OpId from leader: term: 108 index: 99. (term mismatch)
I20260317 21:03:19.933624 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 98
I20260317 21:03:19.933640 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 107.99
I20260317 21:03:19.933655 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 108 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:19.933904 30126 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20260317 21:03:19.935057 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 108 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 106 index: 98. Preceding OpId from leader: term: 108 index: 99. (index mismatch)
I20260317 21:03:19.935251 30134 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20260317 21:03:20.216054 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 108 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:20.216212 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 108 FOLLOWER]: Advancing to term 109
I20260317 21:03:20.217026 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.217257 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 109 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:20.217468 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 109 candidate_status { last_received { term: 108 index: 99 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:20.217468 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 109 candidate_status { last_received { term: 108 index: 99 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:20.217697 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 109 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:20.217824 30113 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 109 FOLLOWER]: Leader election lost for term 109. Reason: could not achieve majority
I20260317 21:03:20.228996 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 109 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 108. Current term is 109. Ops: []
I20260317 21:03:20.229365 30126 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 108.99, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20260317 21:03:20.229501 30126 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 108 LEADER]: Stepping down as leader of term 108
I20260317 21:03:20.229522 30126 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 108 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:20.229593 30126 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 108.99, Last appended by leader: 99, Current term: 108, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.229671 30126 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 108 FOLLOWER]: Advancing to term 109
I20260317 21:03:20.308518 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 108 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:20.308629 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 108 FOLLOWER]: Advancing to term 109
I20260317 21:03:20.309361 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.309509 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 109 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:20.309821 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 109 candidate_status { last_received { term: 108 index: 99 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:20.309859 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 109 candidate_status { last_received { term: 108 index: 99 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:20.309940 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 109 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 109: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:20.310530 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 109.
I20260317 21:03:20.310750 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 109 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:20.310856 30128 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 109 FOLLOWER]: Leader election won for term 109
I20260317 21:03:20.310910 30128 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 109 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:20.310961 30128 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 108.99, Last appended by leader: 99, Current term: 109, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.311503 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 108 to 109, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 109 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:20.341476 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 109 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:20.341578 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 109 FOLLOWER]: Advancing to term 110
I20260317 21:03:20.342206 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.342381 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 110 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:20.342638 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 110 candidate_status { last_received { term: 108 index: 99 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:20.342715 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 109 FOLLOWER]: Advancing to term 110
I20260317 21:03:20.342604 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 110 candidate_status { last_received { term: 108 index: 99 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:20.343324 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 110 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 110.
I20260317 21:03:20.343528 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 110 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:20.343654 30113 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 110 FOLLOWER]: Leader election won for term 110
I20260317 21:03:20.343735 30113 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 110 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:20.343786 30113 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 108.99, Last appended by leader: 99, Current term: 110, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.344292 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 109 to 110, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 110 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:20.403038 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 110 LEADER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 109. Current term is 110. Ops: []
I20260317 21:03:20.403370 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20260317 21:03:20.403519 30128 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 110.100 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 99
I20260317 21:03:20.403565 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 109 LEADER]: Stepping down as leader of term 109
I20260317 21:03:20.403594 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 109 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:20.403709 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 109.100, Last appended by leader: 100, Current term: 109, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.403851 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 109 FOLLOWER]: Advancing to term 110
I20260317 21:03:20.424988 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 110 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 108 index: 99. Preceding OpId from leader: term: 110 index: 100. (index mismatch)
I20260317 21:03:20.425364 30113 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20260317 21:03:20.426528 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 110 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 109 index: 100. Preceding OpId from leader: term: 110 index: 100. (term mismatch)
I20260317 21:03:20.426582 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 99
I20260317 21:03:20.426601 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 109.100
I20260317 21:03:20.426618 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 110 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:20.426774 30133 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20260317 21:03:20.527352 30126 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 110 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:20.527477 30126 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 110 FOLLOWER]: Advancing to term 111
I20260317 21:03:20.528187 30126 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.528373 30126 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 111 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:20.528677 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 111 candidate_status { last_received { term: 110 index: 100 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:20.528707 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 111 candidate_status { last_received { term: 110 index: 100 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:20.528800 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 110 FOLLOWER]: Advancing to term 111
I20260317 21:03:20.529429 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 111 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 111.
I20260317 21:03:20.529630 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 111 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:20.529733 30126 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 111 FOLLOWER]: Leader election won for term 111
I20260317 21:03:20.529779 30126 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 111 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:20.529831 30126 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 110.100, Last appended by leader: 100, Current term: 111, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.530330 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 110 to 111, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 111 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:20.535812 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 111 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 110. Current term is 111. Ops: []
I20260317 21:03:20.536085 30135 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 110.100, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20260317 21:03:20.536216 30135 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 110 LEADER]: Stepping down as leader of term 110
I20260317 21:03:20.536253 30135 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 110 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:20.536319 30135 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 110.100, Last appended by leader: 100, Current term: 110, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.536424 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 110 FOLLOWER]: Advancing to term 111
I20260317 21:03:20.612190 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 111 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 110 index: 100. Preceding OpId from leader: term: 111 index: 101. (index mismatch)
I20260317 21:03:20.612579 30126 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20260317 21:03:20.613803 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 111 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 110 index: 100. Preceding OpId from leader: term: 111 index: 101. (index mismatch)
I20260317 21:03:20.614027 30136 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20260317 21:03:20.714771 30132 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 111 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:20.714884 30132 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 111 FOLLOWER]: Advancing to term 112
I20260317 21:03:20.716607 30132 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.716876 30132 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 112 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:20.717149 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 112 candidate_status { last_received { term: 111 index: 101 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:20.717309 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 112 candidate_status { last_received { term: 111 index: 101 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:20.717490 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 112 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:20.717605 30132 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 112 FOLLOWER]: Leader election lost for term 112. Reason: could not achieve majority
I20260317 21:03:20.720908 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 112 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 111. Current term is 112. Ops: []
I20260317 21:03:20.721261 30136 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 111.101, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20260317 21:03:20.721397 30136 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 111 LEADER]: Stepping down as leader of term 111
I20260317 21:03:20.721422 30136 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 111 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:20.721493 30136 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 111.101, Last appended by leader: 101, Current term: 111, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.721592 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 111 FOLLOWER]: Advancing to term 112
I20260317 21:03:20.800082 30135 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 111 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:20.800194 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 111 FOLLOWER]: Advancing to term 112
I20260317 21:03:20.800880 30135 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.801110 30135 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 112 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:20.801337 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 112 candidate_status { last_received { term: 111 index: 101 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:20.801386 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 112 candidate_status { last_received { term: 111 index: 101 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:20.801447 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 112 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 112: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:20.801975 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 112 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 112.
I20260317 21:03:20.802197 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 112 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:20.802305 30135 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 112 FOLLOWER]: Leader election won for term 112
I20260317 21:03:20.802361 30135 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 112 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:20.802418 30135 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 111.101, Last appended by leader: 101, Current term: 112, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.802944 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 111 to 112, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 112 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:20.845957 30132 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 112 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:20.846074 30132 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 112 FOLLOWER]: Advancing to term 113
I20260317 21:03:20.846799 30132 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.846987 30132 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 113 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:20.847316 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 113 candidate_status { last_received { term: 111 index: 101 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:20.847342 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 113 candidate_status { last_received { term: 111 index: 101 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:20.847493 29887 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 112 FOLLOWER]: Advancing to term 113
I20260317 21:03:20.848040 29887 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 113.
I20260317 21:03:20.848291 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 113 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:20.848445 30132 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 113 FOLLOWER]: Leader election won for term 113
I20260317 21:03:20.848527 30132 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 113 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:20.848579 30132 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 111.101, Last appended by leader: 101, Current term: 113, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.849022 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 112 to 113, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 113 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:20.893864 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:20.919703 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 113 LEADER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 112. Current term is 113. Ops: []
I20260317 21:03:20.920032 30135 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20260317 21:03:20.920125 30135 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 113.102 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 101
I20260317 21:03:20.920147 30135 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 112 LEADER]: Stepping down as leader of term 112
I20260317 21:03:20.920162 30135 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 112 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:20.920229 30135 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 112.102, Last appended by leader: 102, Current term: 112, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:20.920312 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 112 FOLLOWER]: Advancing to term 113
I20260317 21:03:20.959713 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 113 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 112 index: 102. Preceding OpId from leader: term: 113 index: 102. (term mismatch)
I20260317 21:03:20.959794 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 101
I20260317 21:03:20.959808 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 112.102
I20260317 21:03:20.959823 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 113 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:20.960120 30132 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20260317 21:03:20.961380 29887 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 113 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 111 index: 101. Preceding OpId from leader: term: 113 index: 102. (index mismatch)
I20260317 21:03:20.961582 30132 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20260317 21:03:21.062552 30136 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 113 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:21.062674 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 113 FOLLOWER]: Advancing to term 114
I20260317 21:03:21.063426 30136 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.063647 30136 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 114 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:21.063870 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 114 candidate_status { last_received { term: 113 index: 102 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:21.063884 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 114 candidate_status { last_received { term: 113 index: 102 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:21.064167 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 114 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:21.064258 30136 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 114 FOLLOWER]: Leader election lost for term 114. Reason: could not achieve majority
I20260317 21:03:21.073091 29887 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 114 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 113. Current term is 114. Ops: []
I20260317 21:03:21.073454 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 113.102, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20260317 21:03:21.073562 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 113 LEADER]: Stepping down as leader of term 113
I20260317 21:03:21.073580 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 113 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:21.073645 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 113.102, Last appended by leader: 102, Current term: 113, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.073734 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 113 FOLLOWER]: Advancing to term 114
I20260317 21:03:21.149250 30135 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 113 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:21.149374 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 113 FOLLOWER]: Advancing to term 114
I20260317 21:03:21.150151 30135 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.150359 30135 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 114 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:21.150679 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 114 candidate_status { last_received { term: 113 index: 102 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:21.150679 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 114 candidate_status { last_received { term: 113 index: 102 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:21.150942 29887 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 114 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 114: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:21.151517 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 114 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 114.
I20260317 21:03:21.151767 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 114 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:21.151934 30135 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 114 FOLLOWER]: Leader election won for term 114
I20260317 21:03:21.152031 30135 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 114 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:21.152113 30135 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 113.102, Last appended by leader: 102, Current term: 114, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.152647 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 113 to 114, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 114 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:21.183542 30136 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 114 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:21.183650 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 114 FOLLOWER]: Advancing to term 115
I20260317 21:03:21.184350 30136 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.184557 30136 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 115 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:21.184790 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 115 candidate_status { last_received { term: 113 index: 102 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:21.184819 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 115 candidate_status { last_received { term: 113 index: 102 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:21.184919 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 114 FOLLOWER]: Advancing to term 115
I20260317 21:03:21.185619 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 115.
I20260317 21:03:21.185838 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 115 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:21.185931 30136 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 115 FOLLOWER]: Leader election won for term 115
I20260317 21:03:21.185992 30136 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 115 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:21.186048 30136 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 113.102, Last appended by leader: 102, Current term: 115, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.186573 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 114 to 115, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 115 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:21.261996 29961 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 114 LEADER]: Stepping down as leader of term 114
I20260317 21:03:21.262077 29961 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 114 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:21.262164 29961 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 114.103, Last appended by leader: 103, Current term: 114, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.262262 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 114 FOLLOWER]: Advancing to term 115
I20260317 21:03:21.262874 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 115 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 114 index: 103. Preceding OpId from leader: term: 115 index: 103. (term mismatch)
I20260317 21:03:21.262912 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 102
I20260317 21:03:21.262925 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 114.103
I20260317 21:03:21.262939 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 115 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:21.263197 30136 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20260317 21:03:21.264189 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 115 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 113 index: 102. Preceding OpId from leader: term: 115 index: 103. (index mismatch)
I20260317 21:03:21.264408 30136 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20260317 21:03:21.365744 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 115 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:21.365896 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 115 FOLLOWER]: Advancing to term 116
I20260317 21:03:21.366606 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.366796 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 116 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:21.367086 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 116 candidate_status { last_received { term: 115 index: 103 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:21.367133 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 116 candidate_status { last_received { term: 115 index: 103 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:21.367342 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 116 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:21.367468 30128 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 116 FOLLOWER]: Leader election lost for term 116. Reason: could not achieve majority
I20260317 21:03:21.379331 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 116 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 115. Current term is 116. Ops: []
I20260317 21:03:21.379695 30126 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 115.103, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20260317 21:03:21.379822 30126 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 115 LEADER]: Stepping down as leader of term 115
I20260317 21:03:21.379877 30126 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 115 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:21.379945 30126 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 115.103, Last appended by leader: 103, Current term: 115, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.380043 30126 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 115 FOLLOWER]: Advancing to term 116
I20260317 21:03:21.487761 30135 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 115 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:21.487911 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 116 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:21.487931 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 115 FOLLOWER]: Advancing to term 116
I20260317 21:03:21.488082 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 116 FOLLOWER]: Advancing to term 117
I20260317 21:03:21.488780 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.488775 30135 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.489012 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 117 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:21.489059 30135 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 116 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:21.489252 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 117 candidate_status { last_received { term: 115 index: 103 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:21.489271 29887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 116 candidate_status { last_received { term: 115 index: 103 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:21.489230 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 117 candidate_status { last_received { term: 115 index: 103 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:21.489333 29886 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 116 FOLLOWER]: Advancing to term 117
I20260317 21:03:21.489355 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 116 FOLLOWER]: Advancing to term 117
I20260317 21:03:21.489244 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 116 candidate_status { last_received { term: 115 index: 103 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:21.489562 29811 raft_consensus.cc:2368] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 117 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 116. Current term is 117.
I20260317 21:03:21.489697 29928 leader_election.cc:400] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 116 election: Vote denied by peer 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093) with higher term. Message: Invalid argument: T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 117 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 116. Current term is 117.
I20260317 21:03:21.489744 29928 leader_election.cc:403] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 116 election: Cancelling election due to peer responding with higher term
I20260317 21:03:21.490090 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 117.
I20260317 21:03:21.490088 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 117.
I20260317 21:03:21.490232 30135 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 117 FOLLOWER]: Leader election lost for term 116. Reason: Vote denied by peer 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093) with higher term. Message: Invalid argument: T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 117 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 116. Current term is 117.
I20260317 21:03:21.490316 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 117 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:03:21.490419 30128 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 117 FOLLOWER]: Leader election won for term 117
I20260317 21:03:21.490494 30128 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 117 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:21.490545 30128 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 115.103, Last appended by leader: 103, Current term: 117, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.491004 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 115 to 117, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 117 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:21.586694 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 117 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 115 index: 103. Preceding OpId from leader: term: 117 index: 104. (index mismatch)
I20260317 21:03:21.587118 30128 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20260317 21:03:21.588328 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 117 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 115 index: 103. Preceding OpId from leader: term: 117 index: 104. (index mismatch)
I20260317 21:03:21.588627 30138 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20260317 21:03:21.690008 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 117 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:21.690114 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 117 FOLLOWER]: Advancing to term 118
I20260317 21:03:21.690788 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.690972 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 118 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:21.691205 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 118 candidate_status { last_received { term: 117 index: 104 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:21.691234 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 118 candidate_status { last_received { term: 117 index: 104 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:21.691517 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 118 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:21.691643 30113 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 118 FOLLOWER]: Leader election lost for term 118. Reason: could not achieve majority
I20260317 21:03:21.711000 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 118 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 117. Current term is 118. Ops: []
I20260317 21:03:21.711287 30132 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 117.104, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20260317 21:03:21.711405 30132 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 117 LEADER]: Stepping down as leader of term 117
I20260317 21:03:21.711431 30132 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 117 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:21.711484 30132 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 117.104, Last appended by leader: 104, Current term: 117, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.711565 30132 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 117 FOLLOWER]: Advancing to term 118
I20260317 21:03:21.779225 30126 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 117 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:21.779337 30126 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 117 FOLLOWER]: Advancing to term 118
I20260317 21:03:21.780071 30126 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.780258 30126 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 118 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:21.780503 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 118 candidate_status { last_received { term: 117 index: 104 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:21.780544 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 118 candidate_status { last_received { term: 117 index: 104 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:21.780624 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 118 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 118: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:21.781312 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 118 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 118.
I20260317 21:03:21.781512 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 118 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:21.781610 30126 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 118 FOLLOWER]: Leader election won for term 118
I20260317 21:03:21.781670 30126 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 118 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:21.781736 30126 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 117.104, Last appended by leader: 104, Current term: 118, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.782326 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 117 to 118, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 118 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:21.806654 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 118 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:21.806798 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 118 FOLLOWER]: Advancing to term 119
I20260317 21:03:21.807534 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.807725 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 119 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:21.807940 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 119 candidate_status { last_received { term: 117 index: 104 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:21.808023 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 118 FOLLOWER]: Advancing to term 119
I20260317 21:03:21.808095 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 119 candidate_status { last_received { term: 117 index: 104 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:21.808688 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 119 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 119.
I20260317 21:03:21.808914 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 119 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:21.809022 30113 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 119 FOLLOWER]: Leader election won for term 119
I20260317 21:03:21.809108 30113 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 119 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:21.809186 30113 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 117.104, Last appended by leader: 104, Current term: 119, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.809612 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 118 to 119, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 119 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:21.894155 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:21.894928 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 119 LEADER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 118. Current term is 119. Ops: []
I20260317 21:03:21.895246 30126 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20260317 21:03:21.895473 30126 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 119.105 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 104
I20260317 21:03:21.895550 30136 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 118 LEADER]: Stepping down as leader of term 118
I20260317 21:03:21.895609 30136 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 118 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:21.895680 30136 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 118.105, Last appended by leader: 105, Current term: 118, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:21.895777 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 118 FOLLOWER]: Advancing to term 119
I20260317 21:03:21.908293 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 119 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 118 index: 105. Preceding OpId from leader: term: 119 index: 105. (term mismatch)
I20260317 21:03:21.908396 29886 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 104
I20260317 21:03:21.908416 29886 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 118.105
I20260317 21:03:21.908439 29886 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 119 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:21.908742 30113 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20260317 21:03:21.910264 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 119 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 117 index: 104. Preceding OpId from leader: term: 119 index: 105. (index mismatch)
I20260317 21:03:21.910540 30139 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20260317 21:03:22.011543 30132 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 119 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:22.011652 30132 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 119 FOLLOWER]: Advancing to term 120
I20260317 21:03:22.012548 30132 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.012758 30132 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 120 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:22.013018 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 120 candidate_status { last_received { term: 119 index: 105 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:22.013046 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 120 candidate_status { last_received { term: 119 index: 105 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:22.013139 29886 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 119 FOLLOWER]: Advancing to term 120
I20260317 21:03:22.013849 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 120 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 120.
I20260317 21:03:22.014137 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 120 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:22.014241 30132 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 120 FOLLOWER]: Leader election won for term 120
I20260317 21:03:22.014303 30132 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 120 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:22.014358 30132 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 119.105, Last appended by leader: 105, Current term: 120, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.014946 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 119 to 120, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 120 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:22.021916 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 120 LEADER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 119. Current term is 120. Ops: []
I20260317 21:03:22.022168 30135 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 119.105, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20260317 21:03:22.022310 30135 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 120.106 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 105
I20260317 21:03:22.022349 30139 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 119 LEADER]: Stepping down as leader of term 119
I20260317 21:03:22.022395 30139 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 119 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:22.022480 30139 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 119.105, Last appended by leader: 105, Current term: 119, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.022585 30139 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 119 FOLLOWER]: Advancing to term 120
I20260317 21:03:22.099517 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 120 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 119 index: 105. Preceding OpId from leader: term: 120 index: 106. (index mismatch)
I20260317 21:03:22.099884 30132 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20260317 21:03:22.100944 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 120 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 119 index: 105. Preceding OpId from leader: term: 120 index: 106. (index mismatch)
I20260317 21:03:22.101260 30138 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20260317 21:03:22.201655 30136 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 120 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:22.201763 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 120 FOLLOWER]: Advancing to term 121
I20260317 21:03:22.202605 30136 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.203084 30136 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 121 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:22.203280 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 121 candidate_status { last_received { term: 120 index: 106 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:22.203328 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 121 candidate_status { last_received { term: 120 index: 106 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:22.203387 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 120 FOLLOWER]: Advancing to term 121
I20260317 21:03:22.204118 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 121 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 121.
I20260317 21:03:22.204391 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 121 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:22.204514 30136 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 121 FOLLOWER]: Leader election won for term 121
I20260317 21:03:22.204598 30136 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 121 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:22.204655 30136 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 120.106, Last appended by leader: 106, Current term: 121, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.205147 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 120 to 121, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 121 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:22.209851 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 121 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 120. Current term is 121. Ops: []
I20260317 21:03:22.210204 30138 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 120.106, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20260317 21:03:22.210358 30138 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 120 LEADER]: Stepping down as leader of term 120
I20260317 21:03:22.210386 30138 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 120 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:22.210469 30138 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 120.106, Last appended by leader: 106, Current term: 120, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.210563 30138 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 120 FOLLOWER]: Advancing to term 121
I20260317 21:03:22.296887 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 121 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 120 index: 106. Preceding OpId from leader: term: 121 index: 107. (index mismatch)
I20260317 21:03:22.297219 30136 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20260317 21:03:22.298331 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 121 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 120 index: 106. Preceding OpId from leader: term: 121 index: 107. (index mismatch)
I20260317 21:03:22.298549 30136 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20260317 21:03:22.399520 30139 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 121 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:22.399628 30139 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 121 FOLLOWER]: Advancing to term 122
I20260317 21:03:22.400352 30139 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.400566 30139 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 122 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:22.400832 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 122 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:22.400869 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 122 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:22.400990 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 121 FOLLOWER]: Advancing to term 122
I20260317 21:03:22.401625 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 122 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 122.
I20260317 21:03:22.401847 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 122 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:22.402009 30139 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 122 FOLLOWER]: Leader election won for term 122
I20260317 21:03:22.402105 30139 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 122 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:22.402163 30139 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 121.107, Last appended by leader: 107, Current term: 122, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.402748 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 121 to 122, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 122 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:22.406965 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 122 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 121. Current term is 122. Ops: []
I20260317 21:03:22.407251 30136 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 121.107, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20260317 21:03:22.407347 30136 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 121 LEADER]: Stepping down as leader of term 121
I20260317 21:03:22.407367 30136 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 121 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:22.407433 30136 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 121.107, Last appended by leader: 107, Current term: 121, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.407509 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 121 FOLLOWER]: Advancing to term 122
I20260317 21:03:22.491253 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 122 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 121 index: 107. Preceding OpId from leader: term: 122 index: 108. (index mismatch)
I20260317 21:03:22.491569 30139 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20260317 21:03:22.492588 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 122 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 121 index: 107. Preceding OpId from leader: term: 122 index: 108. (index mismatch)
I20260317 21:03:22.492807 30113 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20260317 21:03:22.859753 30136 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 122 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:22.859864 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 122 FOLLOWER]: Advancing to term 123
I20260317 21:03:22.860566 30136 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.860755 30136 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 123 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:22.861042 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 123 candidate_status { last_received { term: 122 index: 108 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:22.861049 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 123 candidate_status { last_received { term: 122 index: 108 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:22.861356 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 123 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:22.861481 30136 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 123 FOLLOWER]: Leader election lost for term 123. Reason: could not achieve majority
I20260317 21:03:22.872280 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 123 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 122. Current term is 123. Ops: []
I20260317 21:03:22.872632 30135 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 122.108, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20260317 21:03:22.872792 30113 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 122 LEADER]: Stepping down as leader of term 122
I20260317 21:03:22.872828 30113 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 122 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:22.872911 30113 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 122.108, Last appended by leader: 108, Current term: 122, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.873008 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 122 FOLLOWER]: Advancing to term 123
I20260317 21:03:22.894454 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:22.941785 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 122 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:22.941887 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 122 FOLLOWER]: Advancing to term 123
I20260317 21:03:22.942603 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.942776 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 123 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:22.943022 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 123 candidate_status { last_received { term: 122 index: 108 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:22.943029 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 123 candidate_status { last_received { term: 122 index: 108 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:22.943149 29886 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 123 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 123: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:22.943783 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 123 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 123.
I20260317 21:03:22.943996 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 123 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:22.944089 30128 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 123 FOLLOWER]: Leader election won for term 123
I20260317 21:03:22.944146 30128 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 123 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:22.944200 30128 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 122.108, Last appended by leader: 108, Current term: 123, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.944689 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 122 to 123, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 123 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:22.981118 30136 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 123 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:22.981251 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 123 FOLLOWER]: Advancing to term 124
I20260317 21:03:22.981885 30136 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.982075 30136 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 124 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:22.982323 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 124 candidate_status { last_received { term: 122 index: 108 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:22.982354 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 124 candidate_status { last_received { term: 122 index: 108 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:22.982599 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 123 FOLLOWER]: Advancing to term 124
I20260317 21:03:22.983106 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 124 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 124.
I20260317 21:03:22.983284 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 124 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:22.983385 30136 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 124 FOLLOWER]: Leader election won for term 124
I20260317 21:03:22.983528 30136 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 124 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:22.983587 30136 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 122.108, Last appended by leader: 108, Current term: 124, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:22.984026 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 123 to 124, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 124 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:23.032248 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 124 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 123. Current term is 124. Ops: []
I20260317 21:03:23.032559 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20260317 21:03:23.032691 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 123 LEADER]: Stepping down as leader of term 123
I20260317 21:03:23.032722 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 123 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:23.032799 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 123.109, Last appended by leader: 109, Current term: 123, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.032872 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 123 FOLLOWER]: Advancing to term 124
I20260317 21:03:23.092119 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 124 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 123 index: 109. Preceding OpId from leader: term: 124 index: 109. (term mismatch)
I20260317 21:03:23.092204 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 108
I20260317 21:03:23.092242 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 123.109
I20260317 21:03:23.092257 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 124 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:23.092536 30136 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20260317 21:03:23.093613 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 124 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 122 index: 108. Preceding OpId from leader: term: 124 index: 109. (index mismatch)
I20260317 21:03:23.093802 30136 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20260317 21:03:23.195102 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 124 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:23.195212 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 124 FOLLOWER]: Advancing to term 125
I20260317 21:03:23.195945 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.196143 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 125 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:23.196379 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 125 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:23.196382 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 125 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:23.196662 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 125 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:23.196777 30113 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 125 FOLLOWER]: Leader election lost for term 125. Reason: could not achieve majority
I20260317 21:03:23.213935 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 125 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 124. Current term is 125. Ops: []
I20260317 21:03:23.214218 30136 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 124.109, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20260317 21:03:23.214367 30136 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 124 LEADER]: Stepping down as leader of term 124
I20260317 21:03:23.214409 30136 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 124 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:23.214483 30136 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 124.109, Last appended by leader: 109, Current term: 124, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.214560 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 124 FOLLOWER]: Advancing to term 125
I20260317 21:03:23.271793 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 124 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:23.271919 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 124 FOLLOWER]: Advancing to term 125
I20260317 21:03:23.272605 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.272787 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 125 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:23.273015 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 125 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:23.273131 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 125 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 125: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:23.273047 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 125 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:23.273890 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 125 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 125.
I20260317 21:03:23.274060 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 125 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:23.274159 30128 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 125 FOLLOWER]: Leader election won for term 125
I20260317 21:03:23.274212 30128 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 125 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:23.274256 30128 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 124.109, Last appended by leader: 109, Current term: 125, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.274763 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 124 to 125, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 125 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:23.333803 30113 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 125 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:23.333930 30113 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 125 FOLLOWER]: Advancing to term 126
I20260317 21:03:23.334622 30113 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.334832 30113 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 126 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:23.335062 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 126 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:23.335062 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 126 candidate_status { last_received { term: 124 index: 109 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:23.335227 29886 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 125 FOLLOWER]: Advancing to term 126
I20260317 21:03:23.335872 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 126 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 126.
I20260317 21:03:23.336073 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 126 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:23.336179 30113 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 126 FOLLOWER]: Leader election won for term 126
I20260317 21:03:23.336285 30113 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 126 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:23.336330 30113 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 124.109, Last appended by leader: 109, Current term: 126, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.336807 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 125 to 126, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 126 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:23.364418 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 126 LEADER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 125. Current term is 126. Ops: []
I20260317 21:03:23.364748 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20260317 21:03:23.364881 30128 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 126.110 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 109
I20260317 21:03:23.364929 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 125 LEADER]: Stepping down as leader of term 125
I20260317 21:03:23.364961 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 125 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:23.365032 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 125.110, Last appended by leader: 110, Current term: 125, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.365115 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 125 FOLLOWER]: Advancing to term 126
I20260317 21:03:23.413375 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 126 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 124 index: 109. Preceding OpId from leader: term: 126 index: 110. (index mismatch)
I20260317 21:03:23.413733 30113 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20260317 21:03:23.414898 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 126 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 125 index: 110. Preceding OpId from leader: term: 126 index: 110. (term mismatch)
I20260317 21:03:23.414963 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 109
I20260317 21:03:23.414987 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 125.110
I20260317 21:03:23.415022 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 126 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:23.415261 30113 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20260317 21:03:23.626751 30138 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 126 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:23.626857 30138 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 126 FOLLOWER]: Advancing to term 127
I20260317 21:03:23.627727 30138 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.627921 30138 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 127 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:23.628160 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 127 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:23.628192 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 127 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:23.628396 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 127 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:23.628500 30138 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 127 FOLLOWER]: Leader election lost for term 127. Reason: could not achieve majority
I20260317 21:03:23.638928 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 127 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 126. Current term is 127. Ops: []
I20260317 21:03:23.639247 30135 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 126.110, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20260317 21:03:23.639353 30135 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 126 LEADER]: Stepping down as leader of term 126
I20260317 21:03:23.639375 30135 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 126 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:23.639473 30135 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 126.110, Last appended by leader: 110, Current term: 126, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.639562 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 126 FOLLOWER]: Advancing to term 127
I20260317 21:03:23.689767 30136 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 126 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:23.689888 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 126 FOLLOWER]: Advancing to term 127
I20260317 21:03:23.690675 30136 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.690877 30136 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 127 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:23.691095 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 127 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:23.691104 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 127 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:23.691298 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 127 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 127: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:23.691813 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 127 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 127.
I20260317 21:03:23.692056 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 127 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:23.692160 30136 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 127 FOLLOWER]: Leader election won for term 127
I20260317 21:03:23.692219 30136 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 127 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:23.692272 30136 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 126.110, Last appended by leader: 110, Current term: 127, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.692898 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 126 to 127, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 127 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:23.746006 30138 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 127 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:23.746109 30138 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 127 FOLLOWER]: Advancing to term 128
I20260317 21:03:23.746794 30138 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.746932 30138 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 128 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:23.747255 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 128 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:23.747332 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 128 candidate_status { last_received { term: 126 index: 110 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:23.747357 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 127 FOLLOWER]: Advancing to term 128
I20260317 21:03:23.748117 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 128 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 128.
I20260317 21:03:23.748400 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 128 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:23.748562 30138 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 128 FOLLOWER]: Leader election won for term 128
I20260317 21:03:23.748656 30138 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 128 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:23.748749 30138 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 126.110, Last appended by leader: 110, Current term: 128, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.749303 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 127 to 128, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 128 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:23.771289 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 128 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 127. Current term is 128. Ops: []
I20260317 21:03:23.771737 30136 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20260317 21:03:23.771883 30136 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 127 LEADER]: Stepping down as leader of term 127
I20260317 21:03:23.771921 30136 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 127 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:23.771991 30136 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 127.111, Last appended by leader: 111, Current term: 127, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:23.772081 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 127 FOLLOWER]: Advancing to term 128
I20260317 21:03:23.847812 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 128 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 126 index: 110. Preceding OpId from leader: term: 128 index: 111. (index mismatch)
I20260317 21:03:23.848187 30138 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20260317 21:03:23.849294 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 128 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 127 index: 111. Preceding OpId from leader: term: 128 index: 111. (term mismatch)
I20260317 21:03:23.849352 29886 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 110
I20260317 21:03:23.849375 29886 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 127.111
I20260317 21:03:23.849393 29886 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 128 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:23.849578 30144 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20260317 21:03:23.894737 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:24.127058 30136 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 128 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:24.127190 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 128 FOLLOWER]: Advancing to term 129
I20260317 21:03:24.127964 30136 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.128175 30136 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 129 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:24.128453 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 129 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:24.128472 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 129 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:24.128751 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 129 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:24.128867 30136 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 129 FOLLOWER]: Leader election lost for term 129. Reason: could not achieve majority
I20260317 21:03:24.149971 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 129 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 128. Current term is 129. Ops: []
I20260317 21:03:24.150343 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 128.111, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20260317 21:03:24.150494 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 128 LEADER]: Stepping down as leader of term 128
I20260317 21:03:24.150540 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 128 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:24.150606 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 128.111, Last appended by leader: 111, Current term: 128, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.150686 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 128 FOLLOWER]: Advancing to term 129
I20260317 21:03:24.202988 30135 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 128 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:24.203106 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 128 FOLLOWER]: Advancing to term 129
I20260317 21:03:24.203821 30135 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.204015 30135 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 129 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:24.204241 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 129 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:24.204309 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 129 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:24.204398 29886 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 129: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:24.205044 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 129.
I20260317 21:03:24.205289 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 129 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:24.205533 30135 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 129 FOLLOWER]: Leader election won for term 129
I20260317 21:03:24.205647 30135 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 129 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:24.205693 30135 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 128.111, Last appended by leader: 111, Current term: 129, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.206264 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 128 to 129, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 129 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:24.229547 30136 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 129 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:24.229650 30136 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 129 FOLLOWER]: Advancing to term 130
I20260317 21:03:24.230315 30136 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.230506 30136 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 130 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:24.230736 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 130 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:24.230742 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 130 candidate_status { last_received { term: 128 index: 111 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:24.230844 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 129 FOLLOWER]: Advancing to term 130
I20260317 21:03:24.231463 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 130 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 130.
I20260317 21:03:24.231722 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 130 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:24.231832 30136 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 130 FOLLOWER]: Leader election won for term 130
I20260317 21:03:24.231905 30136 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 130 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:24.231957 30136 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 128.111, Last appended by leader: 111, Current term: 130, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.232502 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 129 to 130, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 130 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:24.303959 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 130 LEADER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 129. Current term is 130. Ops: []
I20260317 21:03:24.304351 30135 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20260317 21:03:24.304458 30135 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 24015e78ad6341a9a2bc13754a55b581 log is divergent from this leader: its last log entry 130.112 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 111
I20260317 21:03:24.304484 30135 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 129 LEADER]: Stepping down as leader of term 129
I20260317 21:03:24.304498 30135 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 129 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:24.304571 30135 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 129.112, Last appended by leader: 112, Current term: 129, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.304668 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 129 FOLLOWER]: Advancing to term 130
I20260317 21:03:24.339987 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 130 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 129 index: 112. Preceding OpId from leader: term: 130 index: 112. (term mismatch)
I20260317 21:03:24.340101 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 111
I20260317 21:03:24.340122 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 129.112
I20260317 21:03:24.340145 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 130 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:24.340479 30136 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20260317 21:03:24.341950 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 130 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 128 index: 111. Preceding OpId from leader: term: 130 index: 112. (index mismatch)
I20260317 21:03:24.342139 30136 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20260317 21:03:24.530004 30135 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 130 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:24.530134 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 130 FOLLOWER]: Advancing to term 131
I20260317 21:03:24.530798 30135 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.530934 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 131 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 130. Current term is 131. Ops: []
I20260317 21:03:24.531046 30135 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 131 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:24.531105 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 131 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:24.531150 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 131 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:24.531200 30146 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 130.112, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20260317 21:03:24.531352 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 131 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:24.531391 30146 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 130 LEADER]: Stepping down as leader of term 130
I20260317 21:03:24.531445 30146 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 130 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:24.531463 30135 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 131 FOLLOWER]: Leader election lost for term 131. Reason: could not achieve majority
I20260317 21:03:24.531512 30146 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 130.112, Last appended by leader: 112, Current term: 130, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.531621 30146 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 130 FOLLOWER]: Advancing to term 131
I20260317 21:03:24.617930 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 130 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:24.618083 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 130 FOLLOWER]: Advancing to term 131
I20260317 21:03:24.618789 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.618999 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 131 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:24.619215 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 131 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:24.619270 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 131 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:24.619320 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 131 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 131: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:24.619896 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 131 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 131.
I20260317 21:03:24.620121 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 131 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:24.620280 30128 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 131 FOLLOWER]: Leader election won for term 131
I20260317 21:03:24.620376 30128 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 131 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:24.620474 30128 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 130.112, Last appended by leader: 112, Current term: 131, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.621033 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 130 to 131, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 131 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:24.662953 30135 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 131 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:24.663057 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 131 FOLLOWER]: Advancing to term 132
I20260317 21:03:24.663712 30135 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.663995 30135 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 132 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:24.664111 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 132 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:24.664111 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 132 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:24.664297 29886 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 131 FOLLOWER]: Advancing to term 132
I20260317 21:03:24.664909 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 132 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 132.
I20260317 21:03:24.665119 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 132 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:24.665297 30135 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 132 FOLLOWER]: Leader election won for term 132
I20260317 21:03:24.665385 30135 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 132 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:24.665448 30135 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 130.112, Last appended by leader: 112, Current term: 132, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.665963 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 131 to 132, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 132 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:24.716312 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 132 LEADER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 131. Current term is 132. Ops: []
I20260317 21:03:24.716732 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20260317 21:03:24.716845 30128 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 132.113 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 112
I20260317 21:03:24.716869 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 131 LEADER]: Stepping down as leader of term 131
I20260317 21:03:24.716883 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 131 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:24.716966 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 131.113, Last appended by leader: 113, Current term: 131, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.717046 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 131 FOLLOWER]: Advancing to term 132
I20260317 21:03:24.767390 30146 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 132 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:24.767536 30146 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 132 FOLLOWER]: Advancing to term 133
I20260317 21:03:24.768304 30146 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.768512 30146 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 133 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:24.768903 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 133 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:24.768906 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 133 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:24.769019 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 132 FOLLOWER]: Advancing to term 133
I20260317 21:03:24.769687 29811 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 for term 133 because replica has last-logged OpId of term: 131 index: 113, which is greater than that of the candidate, which has last-logged OpId of term: 130 index: 112.
I20260317 21:03:24.769918 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 133 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:24.770035 30146 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 133 FOLLOWER]: Leader election lost for term 133. Reason: could not achieve majority
I20260317 21:03:24.785590 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 133 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 132. Current term is 133. Ops: []
I20260317 21:03:24.785903 30135 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20260317 21:03:24.786121 30135 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 131.113 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 112
I20260317 21:03:24.786185 30147 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 132 LEADER]: Stepping down as leader of term 132
I20260317 21:03:24.786293 30147 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 132 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:24.786396 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 133 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 132. Current term is 133. Ops: []
I20260317 21:03:24.786387 30147 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 132.113, Last appended by leader: 113, Current term: 132, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.786548 30147 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 132 FOLLOWER]: Advancing to term 133
I20260317 21:03:24.895023 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:24.919603 30146 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 133 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:24.919724 30146 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 133 FOLLOWER]: Advancing to term 134
I20260317 21:03:24.920511 30146 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:24.920717 30146 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 134 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:24.920975 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 134 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:24.920985 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 134 candidate_status { last_received { term: 130 index: 112 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:24.921070 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 133 FOLLOWER]: Advancing to term 134
I20260317 21:03:24.921118 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 133 FOLLOWER]: Advancing to term 134
I20260317 21:03:24.921722 29811 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 134 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 for term 134 because replica has last-logged OpId of term: 131 index: 113, which is greater than that of the candidate, which has last-logged OpId of term: 130 index: 112.
I20260317 21:03:24.921779 29961 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 134 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 for term 134 because replica has last-logged OpId of term: 132 index: 113, which is greater than that of the candidate, which has last-logged OpId of term: 130 index: 112.
I20260317 21:03:24.922027 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 134 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:24.922132 30146 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 134 FOLLOWER]: Leader election lost for term 134. Reason: could not achieve majority
I20260317 21:03:25.086150 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 134 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:25.086290 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 134 FOLLOWER]: Advancing to term 135
I20260317 21:03:25.087003 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.087174 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 135 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:25.087425 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 135 candidate_status { last_received { term: 131 index: 113 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:25.087457 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 135 candidate_status { last_received { term: 131 index: 113 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:25.087488 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 134 FOLLOWER]: Advancing to term 135
I20260317 21:03:25.087632 29886 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 134 FOLLOWER]: Advancing to term 135
I20260317 21:03:25.088217 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 135 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 135.
I20260317 21:03:25.088326 29961 raft_consensus.cc:2410] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 135 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a for term 135 because replica has last-logged OpId of term: 132 index: 113, which is greater than that of the candidate, which has last-logged OpId of term: 131 index: 113.
I20260317 21:03:25.088512 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 135 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:25.088608 30128 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 135 FOLLOWER]: Leader election won for term 135
I20260317 21:03:25.088692 30128 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 135 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:25.088747 30128 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 131.113, Last appended by leader: 113, Current term: 135, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.089315 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 132 to 135, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 135 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:25.121711 30147 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 135 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:25.121822 30147 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 135 FOLLOWER]: Advancing to term 136
I20260317 21:03:25.122558 30147 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.122747 30147 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 136 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:25.123018 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 136 candidate_status { last_received { term: 132 index: 113 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:25.123023 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 136 candidate_status { last_received { term: 132 index: 113 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:25.123113 29886 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 135 FOLLOWER]: Advancing to term 136
I20260317 21:03:25.123711 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 136.
I20260317 21:03:25.123930 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 136 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:25.124025 30147 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 136 FOLLOWER]: Leader election won for term 136
I20260317 21:03:25.124086 30147 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 136 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:25.124184 30147 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 132.113, Last appended by leader: 113, Current term: 136, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.124684 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 135 to 136, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 136 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:25.174160 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 136 LEADER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 135. Current term is 136. Ops: []
I20260317 21:03:25.174504 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20260317 21:03:25.174640 30128 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 136.114 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 112
I20260317 21:03:25.174690 30144 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 135 LEADER]: Stepping down as leader of term 135
I20260317 21:03:25.174755 30144 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 135 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:25.174814 30144 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 135.114, Last appended by leader: 114, Current term: 135, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.174883 30144 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 135 FOLLOWER]: Advancing to term 136
I20260317 21:03:25.203044 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 136 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 135 index: 114. Preceding OpId from leader: term: 136 index: 114. (term mismatch)
I20260317 21:03:25.203130 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 113
I20260317 21:03:25.203145 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 135.114
I20260317 21:03:25.203161 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 136 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:25.203439 30147 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20260317 21:03:25.203562 30147 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 131.113 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 112
I20260317 21:03:25.203805 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 112
I20260317 21:03:25.203841 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 131.113
I20260317 21:03:25.203855 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 136 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:25.204895 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 136 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 130 index: 112. Preceding OpId from leader: term: 136 index: 114. (index mismatch)
I20260317 21:03:25.205104 30135 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20260317 21:03:25.549777 30148 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 136 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:25.549929 30148 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 136 FOLLOWER]: Advancing to term 137
I20260317 21:03:25.550722 30148 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.550956 30148 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 137 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:25.551188 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 137 candidate_status { last_received { term: 136 index: 114 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:25.551210 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 137 candidate_status { last_received { term: 136 index: 114 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:25.551496 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 137 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:25.551633 30148 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 137 FOLLOWER]: Leader election lost for term 137. Reason: could not achieve majority
I20260317 21:03:25.559116 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 137 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 136. Current term is 137. Ops: []
I20260317 21:03:25.559468 30147 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 136.114, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20260317 21:03:25.559610 30135 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 136 LEADER]: Stepping down as leader of term 136
I20260317 21:03:25.559651 30135 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 136 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:25.559736 30135 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 136.114, Last appended by leader: 114, Current term: 136, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.559830 30135 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 136 FOLLOWER]: Advancing to term 137
I20260317 21:03:25.639206 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 136 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:25.639338 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 136 FOLLOWER]: Advancing to term 137
I20260317 21:03:25.639997 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.640194 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 137 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:25.640470 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 137 candidate_status { last_received { term: 136 index: 114 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:25.640517 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 137 candidate_status { last_received { term: 136 index: 114 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:25.640648 29886 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 137 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 137: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:25.641288 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 137 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 137.
I20260317 21:03:25.641533 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 137 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:25.641644 30128 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 137 FOLLOWER]: Leader election won for term 137
I20260317 21:03:25.641707 30128 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 137 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:25.641768 30128 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 136.114, Last appended by leader: 114, Current term: 137, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.642271 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 136 to 137, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 137 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:25.671193 30148 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 137 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:25.671303 30148 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 137 FOLLOWER]: Advancing to term 138
I20260317 21:03:25.672425 30148 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.672627 30148 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 138 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:25.672843 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 138 candidate_status { last_received { term: 136 index: 114 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:25.673396 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 138 candidate_status { last_received { term: 136 index: 114 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:25.673493 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 137 FOLLOWER]: Advancing to term 138
I20260317 21:03:25.674677 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 138 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 138.
I20260317 21:03:25.674911 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 138 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:25.675028 30148 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 138 FOLLOWER]: Leader election won for term 138
I20260317 21:03:25.675092 30148 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 138 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:25.675143 30148 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 136.114, Last appended by leader: 114, Current term: 138, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.675724 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 137 to 138, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 138 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:25.760035 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 138 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 137. Current term is 138. Ops: []
I20260317 21:03:25.760383 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20260317 21:03:25.760545 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 137 LEADER]: Stepping down as leader of term 137
I20260317 21:03:25.760596 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 137 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:25.760672 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 137.115, Last appended by leader: 115, Current term: 137, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.760753 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 137 FOLLOWER]: Advancing to term 138
I20260317 21:03:25.774331 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 138 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 137 index: 115. Preceding OpId from leader: term: 138 index: 115. (term mismatch)
I20260317 21:03:25.774410 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 114
I20260317 21:03:25.774423 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 137.115
I20260317 21:03:25.774438 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 138 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:25.774680 30146 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20260317 21:03:25.775629 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 138 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 136 index: 114. Preceding OpId from leader: term: 138 index: 115. (index mismatch)
I20260317 21:03:25.775878 30152 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20260317 21:03:25.877296 30147 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 138 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:25.877424 30147 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 138 FOLLOWER]: Advancing to term 139
I20260317 21:03:25.878623 30147 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.878834 30147 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 139 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:25.879060 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 139 candidate_status { last_received { term: 138 index: 115 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:25.879076 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 139 candidate_status { last_received { term: 138 index: 115 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:25.879289 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 139 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:25.879410 30147 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 139 FOLLOWER]: Leader election lost for term 139. Reason: could not achieve majority
I20260317 21:03:25.881897 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 139 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 138. Current term is 139. Ops: []
I20260317 21:03:25.882099 30152 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 138.115, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20260317 21:03:25.882284 30152 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 138 LEADER]: Stepping down as leader of term 138
I20260317 21:03:25.882328 30152 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 138 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:25.882393 30152 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 138.115, Last appended by leader: 115, Current term: 138, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.882494 30152 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 138 FOLLOWER]: Advancing to term 139
I20260317 21:03:25.895380 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:25.986212 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 138 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:25.986320 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 138 FOLLOWER]: Advancing to term 139
I20260317 21:03:25.987047 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.987182 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 139 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:25.987536 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 139 candidate_status { last_received { term: 138 index: 115 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:25.987550 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 139 candidate_status { last_received { term: 138 index: 115 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:25.987632 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 139 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 139: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:25.988196 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 139 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 139.
I20260317 21:03:25.988390 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 139 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:25.988494 30128 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 139 FOLLOWER]: Leader election won for term 139
I20260317 21:03:25.988546 30128 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 139 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:25.988597 30128 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 138.115, Last appended by leader: 115, Current term: 139, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:25.989073 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 138 to 139, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 139 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:26.016997 30147 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 139 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:26.017108 30147 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 139 FOLLOWER]: Advancing to term 140
I20260317 21:03:26.017850 30147 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 140 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.018041 30147 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 140 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:26.018276 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 140 candidate_status { last_received { term: 138 index: 115 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:26.018322 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 140 candidate_status { last_received { term: 138 index: 115 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:26.018396 29886 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 139 FOLLOWER]: Advancing to term 140
I20260317 21:03:26.019021 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 140 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 140.
I20260317 21:03:26.019241 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 140 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:26.019347 30147 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 140 FOLLOWER]: Leader election won for term 140
I20260317 21:03:26.019423 30147 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 140 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:26.019594 30147 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 138.115, Last appended by leader: 115, Current term: 140, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.020258 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 139 to 140, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 140 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:26.080642 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 140 LEADER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 139. Current term is 140. Ops: []
I20260317 21:03:26.080969 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20260317 21:03:26.081087 30128 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer 0b4ef85c36ee46d8bea7064e0759a98a log is divergent from this leader: its last log entry 140.116 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 115
I20260317 21:03:26.081111 30128 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 139 LEADER]: Stepping down as leader of term 139
I20260317 21:03:26.081125 30128 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 139 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:26.081238 30128 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 139.116, Last appended by leader: 116, Current term: 139, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.081312 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 139 FOLLOWER]: Advancing to term 140
I20260317 21:03:26.116087 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 140 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 138 index: 115. Preceding OpId from leader: term: 140 index: 116. (index mismatch)
I20260317 21:03:26.116402 30147 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20260317 21:03:26.117623 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 140 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 139 index: 116. Preceding OpId from leader: term: 140 index: 116. (term mismatch)
I20260317 21:03:26.117682 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 115
I20260317 21:03:26.117702 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 139.116
I20260317 21:03:26.117719 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 140 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:26.117901 30147 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20260317 21:03:26.307191 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 140 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:26.307344 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 140 FOLLOWER]: Advancing to term 141
I20260317 21:03:26.308252 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.308493 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 141 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:26.308748 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 141 candidate_status { last_received { term: 140 index: 116 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:26.308758 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 141 candidate_status { last_received { term: 140 index: 116 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:26.309012 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 141 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:26.309126 30128 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 141 FOLLOWER]: Leader election lost for term 141. Reason: could not achieve majority
I20260317 21:03:26.314481 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 141 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 140. Current term is 141. Ops: []
I20260317 21:03:26.314751 30135 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 140.116, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20260317 21:03:26.314898 30147 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 140 LEADER]: Stepping down as leader of term 140
I20260317 21:03:26.314930 30147 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 140 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:26.315006 30147 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 140.116, Last appended by leader: 116, Current term: 140, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.315091 30147 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 140 FOLLOWER]: Advancing to term 141
I20260317 21:03:26.390352 30152 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 140 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:26.390484 30152 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 140 FOLLOWER]: Advancing to term 141
I20260317 21:03:26.391125 30152 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.391353 30152 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 141 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:26.391568 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 141 candidate_status { last_received { term: 140 index: 116 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:26.391571 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 141 candidate_status { last_received { term: 140 index: 116 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:26.391700 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 141 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 141: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:26.392299 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 141 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 141.
I20260317 21:03:26.392503 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 141 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:26.392601 30152 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 141 FOLLOWER]: Leader election won for term 141
I20260317 21:03:26.392714 30152 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 141 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:26.392758 30152 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 140.116, Last appended by leader: 116, Current term: 141, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.393289 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 140 to 141, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 141 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:26.445830 30128 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 141 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:26.445950 30128 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 141 FOLLOWER]: Advancing to term 142
I20260317 21:03:26.446702 30128 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 142 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.446875 30128 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 142 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:26.447163 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 142 candidate_status { last_received { term: 140 index: 116 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:26.447163 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 142 candidate_status { last_received { term: 140 index: 116 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:26.447335 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 141 FOLLOWER]: Advancing to term 142
I20260317 21:03:26.447933 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 142 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 142.
I20260317 21:03:26.448190 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 142 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:26.448299 30128 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 142 FOLLOWER]: Leader election won for term 142
I20260317 21:03:26.448364 30128 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 142 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:26.448434 30128 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 140.116, Last appended by leader: 116, Current term: 142, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.448936 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 141 to 142, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 142 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:26.491518 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 142 LEADER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 141. Current term is 142. Ops: []
I20260317 21:03:26.491942 30152 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20260317 21:03:26.492093 30152 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 142.117 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 116
I20260317 21:03:26.492132 30153 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 141 LEADER]: Stepping down as leader of term 141
I20260317 21:03:26.492178 30153 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 141 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:26.492269 30153 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 141.117, Last appended by leader: 117, Current term: 141, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.492377 30153 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 141 FOLLOWER]: Advancing to term 142
I20260317 21:03:26.541564 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 142 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 141 index: 117. Preceding OpId from leader: term: 142 index: 117. (term mismatch)
I20260317 21:03:26.541643 29886 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 116
I20260317 21:03:26.541658 29886 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 141.117
I20260317 21:03:26.541673 29886 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 142 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:26.541945 30128 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20260317 21:03:26.543360 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 142 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 140 index: 116. Preceding OpId from leader: term: 142 index: 117. (index mismatch)
I20260317 21:03:26.543545 30128 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20260317 21:03:26.739967 30153 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 142 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:26.740082 30153 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 142 FOLLOWER]: Advancing to term 143
I20260317 21:03:26.740849 30153 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.741150 30153 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 143 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:26.741263 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 143 candidate_status { last_received { term: 142 index: 117 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:26.741302 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 143 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 142. Current term is 143. Ops: []
I20260317 21:03:26.741297 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 143 candidate_status { last_received { term: 142 index: 117 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:26.741528 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 143 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:26.741605 30128 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 142.117, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20260317 21:03:26.741647 30153 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 143 FOLLOWER]: Leader election lost for term 143. Reason: could not achieve majority
I20260317 21:03:26.741724 30154 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 142 LEADER]: Stepping down as leader of term 142
I20260317 21:03:26.741755 30154 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 142 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:26.741838 30154 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 142.117, Last appended by leader: 117, Current term: 142, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.741868 30147 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 142 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:26.741926 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 142 FOLLOWER]: Advancing to term 143
I20260317 21:03:26.741928 30147 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 142 FOLLOWER]: Advancing to term 143
I20260317 21:03:26.742530 30147 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.742686 30147 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 143 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:26.742822 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 143 candidate_status { last_received { term: 142 index: 117 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:26.742873 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 143 candidate_status { last_received { term: 142 index: 117 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:26.742919 29886 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 143 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 143: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:26.743516 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 143 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 143.
I20260317 21:03:26.743722 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 143 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:26.743840 30147 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 143 FOLLOWER]: Leader election won for term 143
I20260317 21:03:26.743942 30147 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 143 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:26.743995 30147 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 142.117, Last appended by leader: 117, Current term: 143, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.744395 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 142 to 143, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 143 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:26.852082 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 143 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 142 index: 117. Preceding OpId from leader: term: 143 index: 118. (index mismatch)
I20260317 21:03:26.852082 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 143 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 142 index: 117. Preceding OpId from leader: term: 143 index: 118. (index mismatch)
I20260317 21:03:26.852416 30147 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20260317 21:03:26.852520 30135 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20260317 21:03:26.895696 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:26.954473 30153 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 143 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:26.954579 30153 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 143 FOLLOWER]: Advancing to term 144
I20260317 21:03:26.955350 30153 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.955578 30153 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 144 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:26.955792 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 144 candidate_status { last_received { term: 143 index: 118 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:26.955806 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 144 candidate_status { last_received { term: 143 index: 118 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:26.956092 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 144 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:26.956218 30153 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 144 FOLLOWER]: Leader election lost for term 144. Reason: could not achieve majority
I20260317 21:03:26.963344 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 144 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 143. Current term is 144. Ops: []
I20260317 21:03:26.963647 30155 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 143.118, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20260317 21:03:26.963788 30147 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 143 LEADER]: Stepping down as leader of term 143
I20260317 21:03:26.963832 30147 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 143 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:26.963910 30147 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 143.118, Last appended by leader: 118, Current term: 143, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:26.963990 30147 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 143 FOLLOWER]: Advancing to term 144
I20260317 21:03:27.049019 30154 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 143 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:27.049144 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 143 FOLLOWER]: Advancing to term 144
I20260317 21:03:27.049868 30154 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.050001 30154 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 144 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:27.050429 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 144 candidate_status { last_received { term: 143 index: 118 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:27.050431 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 144 candidate_status { last_received { term: 143 index: 118 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:27.050597 29886 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 144 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 144: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:27.051100 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 144 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 144.
I20260317 21:03:27.051337 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 144 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:27.051450 30154 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 144 FOLLOWER]: Leader election won for term 144
I20260317 21:03:27.051513 30154 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 144 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:27.051589 30154 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 143.118, Last appended by leader: 118, Current term: 144, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.052111 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 143 to 144, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 144 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:27.062705 30153 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 144 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:27.062848 30153 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 144 FOLLOWER]: Advancing to term 145
I20260317 21:03:27.063611 30153 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.063829 30153 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 145 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:27.064075 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 145 candidate_status { last_received { term: 143 index: 118 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:27.064066 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 145 candidate_status { last_received { term: 143 index: 118 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:27.064218 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 144 FOLLOWER]: Advancing to term 145
I20260317 21:03:27.064805 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 145 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 145.
I20260317 21:03:27.065026 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 145 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:27.065145 30153 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 145 FOLLOWER]: Leader election won for term 145
I20260317 21:03:27.065281 30153 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 145 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:27.065327 30153 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 143.118, Last appended by leader: 118, Current term: 145, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.065795 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 144 to 145, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 145 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:27.144109 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 145 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 144. Current term is 145. Ops: []
I20260317 21:03:27.144451 30154 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20260317 21:03:27.144594 30154 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 144 LEADER]: Stepping down as leader of term 144
I20260317 21:03:27.144629 30154 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 144 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:27.144696 30154 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 144.119, Last appended by leader: 119, Current term: 144, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.144771 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 144 FOLLOWER]: Advancing to term 145
I20260317 21:03:27.170411 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 145 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 144 index: 119. Preceding OpId from leader: term: 145 index: 119. (term mismatch)
I20260317 21:03:27.170500 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 118
I20260317 21:03:27.170516 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 144.119
I20260317 21:03:27.170531 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 145 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:27.170904 30153 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20260317 21:03:27.172247 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 145 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 143 index: 118. Preceding OpId from leader: term: 145 index: 119. (index mismatch)
I20260317 21:03:27.172466 30156 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20260317 21:03:27.445397 30155 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 145 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:27.445531 30155 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 145 FOLLOWER]: Advancing to term 146
I20260317 21:03:27.446275 30155 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.446481 30155 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 146 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:27.446610 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 146 candidate_status { last_received { term: 145 index: 119 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:27.446648 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 146 candidate_status { last_received { term: 145 index: 119 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:27.446979 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 146 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:27.447088 30155 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 146 FOLLOWER]: Leader election lost for term 146. Reason: could not achieve majority
I20260317 21:03:27.459522 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 146 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 145. Current term is 146. Ops: []
I20260317 21:03:27.459838 30156 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 145.119, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20260317 21:03:27.459992 30156 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 145 LEADER]: Stepping down as leader of term 145
I20260317 21:03:27.460031 30156 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 145 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:27.460104 30156 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 145.119, Last appended by leader: 119, Current term: 145, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.460237 30156 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 145 FOLLOWER]: Advancing to term 146
I20260317 21:03:27.545846 30154 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 145 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:27.545964 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 145 FOLLOWER]: Advancing to term 146
I20260317 21:03:27.546703 30154 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.546882 30154 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 146 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:27.547197 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 146 candidate_status { last_received { term: 145 index: 119 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:27.547211 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 146 candidate_status { last_received { term: 145 index: 119 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:27.547326 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 146 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 146: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:27.547885 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 146 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 146.
I20260317 21:03:27.548112 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 146 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:27.548216 30154 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 146 FOLLOWER]: Leader election won for term 146
I20260317 21:03:27.548298 30154 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 146 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:27.548381 30154 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 145.119, Last appended by leader: 119, Current term: 146, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.548890 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 145 to 146, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 146 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:27.564240 30155 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 146 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:27.564354 30155 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 146 FOLLOWER]: Advancing to term 147
I20260317 21:03:27.565071 30155 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.565291 30155 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 147 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:27.565523 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 147 candidate_status { last_received { term: 145 index: 119 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:27.565533 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 147 candidate_status { last_received { term: 145 index: 119 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:27.565616 29886 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 146 FOLLOWER]: Advancing to term 147
I20260317 21:03:27.566201 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 147 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 147.
I20260317 21:03:27.566447 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 147 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:27.566591 30155 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 147 FOLLOWER]: Leader election won for term 147
I20260317 21:03:27.566673 30155 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 147 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:27.566730 30155 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 145.119, Last appended by leader: 119, Current term: 147, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.567194 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 146 to 147, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 147 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:27.634320 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 147 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 146. Current term is 147. Ops: []
I20260317 21:03:27.634788 30154 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20260317 21:03:27.634955 30154 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 146 LEADER]: Stepping down as leader of term 146
I20260317 21:03:27.634994 30154 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 146 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:27.635061 30154 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 146.120, Last appended by leader: 120, Current term: 146, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.635159 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 146 FOLLOWER]: Advancing to term 147
I20260317 21:03:27.652735 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 147 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 146 index: 120. Preceding OpId from leader: term: 147 index: 120. (term mismatch)
I20260317 21:03:27.652817 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 119
I20260317 21:03:27.652832 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 146.120
I20260317 21:03:27.652846 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 147 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:27.653133 30155 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20260317 21:03:27.654280 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 147 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 145 index: 119. Preceding OpId from leader: term: 147 index: 120. (index mismatch)
I20260317 21:03:27.654487 30155 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20260317 21:03:27.895970 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:27.940531 30156 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 147 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:27.940635 30156 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 147 FOLLOWER]: Advancing to term 148
I20260317 21:03:27.941397 30156 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.941597 30156 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 148 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:27.941823 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 148 candidate_status { last_received { term: 147 index: 120 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:27.941848 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 148 candidate_status { last_received { term: 147 index: 120 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:27.942066 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 148 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:27.942199 30156 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 148 FOLLOWER]: Leader election lost for term 148. Reason: could not achieve majority
I20260317 21:03:27.951328 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 148 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 147. Current term is 148. Ops: []
I20260317 21:03:27.951726 30155 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 147.120, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20260317 21:03:27.951848 30155 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 147 LEADER]: Stepping down as leader of term 147
I20260317 21:03:27.951867 30155 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 147 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:27.951936 30155 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 147.120, Last appended by leader: 120, Current term: 147, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.952013 30155 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 147 FOLLOWER]: Advancing to term 148
I20260317 21:03:27.971226 30154 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 147 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:27.971333 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 147 FOLLOWER]: Advancing to term 148
I20260317 21:03:27.971990 30154 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.972173 30154 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 148 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:27.972433 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 148 candidate_status { last_received { term: 147 index: 120 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:27.972450 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 148 candidate_status { last_received { term: 147 index: 120 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:27.972563 29886 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 148 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 148: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:27.973279 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 148 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 148.
I20260317 21:03:27.973519 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 148 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:27.973619 30154 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 148 FOLLOWER]: Leader election won for term 148
I20260317 21:03:27.973699 30154 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 148 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:27.973754 30154 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 147.120, Last appended by leader: 120, Current term: 148, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:27.974316 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 147 to 148, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 148 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:28.080243 30156 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 148 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:28.080372 30156 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 148 FOLLOWER]: Advancing to term 149
I20260317 21:03:28.081202 30156 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.081401 30156 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 149 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:28.081660 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 149 candidate_status { last_received { term: 147 index: 120 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:28.081660 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 149 candidate_status { last_received { term: 147 index: 120 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:28.081791 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 148 FOLLOWER]: Advancing to term 149
I20260317 21:03:28.082373 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 149 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 149.
I20260317 21:03:28.082568 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 149 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:28.082671 30156 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 149 FOLLOWER]: Leader election won for term 149
I20260317 21:03:28.082734 30156 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 149 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:28.082795 30156 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 147.120, Last appended by leader: 120, Current term: 149, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.083257 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 148 to 149, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 149 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:28.087358 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 149 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 148. Current term is 149. Ops: []
I20260317 21:03:28.087644 30154 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20260317 21:03:28.087751 30154 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 148 LEADER]: Stepping down as leader of term 148
I20260317 21:03:28.087788 30154 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 148 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:28.087852 30154 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 148.121, Last appended by leader: 121, Current term: 148, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.087952 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 148 FOLLOWER]: Advancing to term 149
I20260317 21:03:28.198954 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 149 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 147 index: 120. Preceding OpId from leader: term: 149 index: 121. (index mismatch)
I20260317 21:03:28.199348 30156 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20260317 21:03:28.200517 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 149 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 148 index: 121. Preceding OpId from leader: term: 149 index: 121. (term mismatch)
I20260317 21:03:28.200591 29811 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting all ops after (but not including) 120
I20260317 21:03:28.200616 29811 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: Aborting uncommitted NO_OP operation due to leader change: 148.121
I20260317 21:03:28.200637 29811 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 149 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:28.200870 30157 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20260317 21:03:28.301229 30155 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 149 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:28.301355 30155 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 149 FOLLOWER]: Advancing to term 150
I20260317 21:03:28.302155 30155 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.302332 30155 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 150 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:28.302421 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 150 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 149. Current term is 150. Ops: []
I20260317 21:03:28.302503 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 150 candidate_status { last_received { term: 149 index: 121 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:28.302659 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 150 candidate_status { last_received { term: 149 index: 121 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:28.302755 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 149 FOLLOWER]: Advancing to term 150
I20260317 21:03:28.302969 30157 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 149.121, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20260317 21:03:28.303090 30157 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 149 LEADER]: Stepping down as leader of term 149
I20260317 21:03:28.303124 30157 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 149 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:28.303189 30157 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 149.121, Last appended by leader: 121, Current term: 149, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.303270 30157 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 149 FOLLOWER]: Advancing to term 150
I20260317 21:03:28.303537 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 150 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 150.
I20260317 21:03:28.303759 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 150 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:28.303860 30155 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 150 FOLLOWER]: Leader election won for term 150
I20260317 21:03:28.303956 30155 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 150 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:28.304009 30155 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 149.121, Last appended by leader: 121, Current term: 150, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.304462 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 149 to 150, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 150 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:28.389951 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 150 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 149 index: 121. Preceding OpId from leader: term: 150 index: 122. (index mismatch)
I20260317 21:03:28.390302 30155 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20260317 21:03:28.391554 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 150 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 149 index: 121. Preceding OpId from leader: term: 150 index: 122. (index mismatch)
I20260317 21:03:28.391889 30158 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20260317 21:03:28.492621 30154 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 150 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:28.492765 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 150 FOLLOWER]: Advancing to term 151
I20260317 21:03:28.493515 30154 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.493721 30154 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 151 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:28.493973 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 151 candidate_status { last_received { term: 150 index: 122 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:28.493973 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 151 candidate_status { last_received { term: 150 index: 122 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:28.494251 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 151 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:28.494383 30154 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 151 FOLLOWER]: Leader election lost for term 151. Reason: could not achieve majority
I20260317 21:03:28.505540 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 151 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 150. Current term is 151. Ops: []
I20260317 21:03:28.505852 30147 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 150.122, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20260317 21:03:28.505991 30158 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 150 LEADER]: Stepping down as leader of term 150
I20260317 21:03:28.506039 30158 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 150 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:28.506119 30158 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 150.122, Last appended by leader: 122, Current term: 150, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.506212 30158 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 150 FOLLOWER]: Advancing to term 151
I20260317 21:03:28.588800 30157 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 150 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:28.588910 30157 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 150 FOLLOWER]: Advancing to term 151
I20260317 21:03:28.589696 30157 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.589978 30157 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 151 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:28.590103 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 151 candidate_status { last_received { term: 150 index: 122 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:28.590103 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 151 candidate_status { last_received { term: 150 index: 122 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:28.590242 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 151 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 151: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:28.590843 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 151 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 151.
I20260317 21:03:28.591023 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 151 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:28.591125 30157 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 151 FOLLOWER]: Leader election won for term 151
I20260317 21:03:28.591212 30157 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 151 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:28.591269 30157 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 150.122, Last appended by leader: 122, Current term: 151, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.591719 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 150 to 151, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 151 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:28.641705 30154 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 151 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:28.641817 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 151 FOLLOWER]: Advancing to term 152
I20260317 21:03:28.642877 30154 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.643046 30154 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 152 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:28.643275 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 152 candidate_status { last_received { term: 150 index: 122 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:28.643353 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 151 FOLLOWER]: Advancing to term 152
I20260317 21:03:28.643307 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 152 candidate_status { last_received { term: 150 index: 122 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:28.644134 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 152 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 152.
I20260317 21:03:28.644336 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 152 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:28.644497 30154 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 152 FOLLOWER]: Leader election won for term 152
I20260317 21:03:28.644568 30154 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 152 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:28.644623 30154 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 150.122, Last appended by leader: 122, Current term: 152, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.645108 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 151 to 152, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 152 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:28.666985 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 152 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 151. Current term is 152. Ops: []
I20260317 21:03:28.667299 30157 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20260317 21:03:28.667461 30157 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 151 LEADER]: Stepping down as leader of term 151
I20260317 21:03:28.667519 30157 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 151 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:28.667613 30157 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 151.123, Last appended by leader: 123, Current term: 151, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.667711 30157 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 151 FOLLOWER]: Advancing to term 152
I20260317 21:03:28.730633 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 152 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 150 index: 122. Preceding OpId from leader: term: 152 index: 123. (index mismatch)
I20260317 21:03:28.731189 30154 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20260317 21:03:28.731557 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 152 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 151 index: 123. Preceding OpId from leader: term: 152 index: 123. (term mismatch)
I20260317 21:03:28.731618 29886 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 122
I20260317 21:03:28.731637 29886 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 151.123
I20260317 21:03:28.731680 29886 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 152 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:28.731871 30128 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20260317 21:03:28.833521 30158 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 152 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:28.833675 30158 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 152 FOLLOWER]: Advancing to term 153
I20260317 21:03:28.834408 30158 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.834668 30158 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 153 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:28.834909 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 153 candidate_status { last_received { term: 152 index: 123 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:28.834903 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 153 candidate_status { last_received { term: 152 index: 123 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:28.835039 29886 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 152 FOLLOWER]: Advancing to term 153
I20260317 21:03:28.835714 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 153 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 153.
I20260317 21:03:28.836038 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 153 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:28.836205 30158 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 153 FOLLOWER]: Leader election won for term 153
I20260317 21:03:28.836288 30158 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 153 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:28.836335 30158 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 152.123, Last appended by leader: 123, Current term: 153, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.836817 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 153 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 152. Current term is 153. Ops: []
I20260317 21:03:28.836894 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 152 to 153, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 153 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:28.837047 30159 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 152.123, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20260317 21:03:28.837208 30154 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 152 LEADER]: Stepping down as leader of term 152
I20260317 21:03:28.837255 30154 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 152 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:28.837313 30154 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 152.123, Last appended by leader: 123, Current term: 152, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:28.837378 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 152 FOLLOWER]: Advancing to term 153
I20260317 21:03:28.896245 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:28.937417 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 153 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 152 index: 123. Preceding OpId from leader: term: 153 index: 124. (index mismatch)
I20260317 21:03:28.937731 30158 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20260317 21:03:28.938745 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 153 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 152 index: 123. Preceding OpId from leader: term: 153 index: 124. (index mismatch)
I20260317 21:03:28.938952 30158 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20260317 21:03:29.133244 30156 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 153 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:29.133396 30156 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 153 FOLLOWER]: Advancing to term 154
I20260317 21:03:29.134162 30156 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.134454 30156 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 154 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:29.134644 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 154 candidate_status { last_received { term: 153 index: 124 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:29.134963 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 154 candidate_status { last_received { term: 153 index: 124 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:29.135555 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 154 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:29.135807 30156 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 154 FOLLOWER]: Leader election lost for term 154. Reason: could not achieve majority
I20260317 21:03:29.144196 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 154 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 153. Current term is 154. Ops: []
I20260317 21:03:29.144539 30147 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 153.124, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20260317 21:03:29.144682 30147 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 153 LEADER]: Stepping down as leader of term 153
I20260317 21:03:29.144833 30147 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 153 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:29.144905 30147 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 153.124, Last appended by leader: 124, Current term: 153, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.145047 30147 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 153 FOLLOWER]: Advancing to term 154
I20260317 21:03:29.153496 30154 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 153 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:29.153612 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 153 FOLLOWER]: Advancing to term 154
I20260317 21:03:29.154397 30154 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.154613 30154 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 154 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:29.154935 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 154 candidate_status { last_received { term: 153 index: 124 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:29.155011 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 154 candidate_status { last_received { term: 153 index: 124 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:29.155121 29886 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 154 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f1d952e629c4010ae48963a9cdee04a in current term 154: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
I20260317 21:03:29.155728 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 154 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 154.
I20260317 21:03:29.156003 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 154 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:29.156185 30154 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 154 FOLLOWER]: Leader election won for term 154
I20260317 21:03:29.156265 30154 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 154 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:29.156327 30154 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 153.124, Last appended by leader: 124, Current term: 154, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.157369 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 153 to 154, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 154 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:29.235489 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 154 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 153 index: 124. Preceding OpId from leader: term: 154 index: 125. (index mismatch)
I20260317 21:03:29.235766 30154 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20260317 21:03:29.236917 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 154 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 153 index: 124. Preceding OpId from leader: term: 154 index: 125. (index mismatch)
I20260317 21:03:29.237111 30154 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20260317 21:03:29.424857 30147 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 154 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:29.424962 30147 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 154 FOLLOWER]: Advancing to term 155
I20260317 21:03:29.425698 30147 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.425894 30147 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 155 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:29.426090 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 155 candidate_status { last_received { term: 154 index: 125 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:29.426090 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 155 candidate_status { last_received { term: 154 index: 125 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:29.426338 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 155 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:29.426481 30147 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 155 FOLLOWER]: Leader election lost for term 155. Reason: could not achieve majority
I20260317 21:03:29.433887 29961 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 155 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 154. Current term is 155. Ops: []
I20260317 21:03:29.434173 30154 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: INVALID_TERM, Last received: 154.125, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20260317 21:03:29.434335 30160 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 154 LEADER]: Stepping down as leader of term 154
I20260317 21:03:29.434376 30160 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 154 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:29.434468 30160 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 154.125, Last appended by leader: 125, Current term: 154, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.434577 30160 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 154 FOLLOWER]: Advancing to term 155
I20260317 21:03:29.513794 30156 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 154 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:29.513948 30156 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 154 FOLLOWER]: Advancing to term 155
I20260317 21:03:29.514678 30156 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.514933 30156 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 155 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:29.515228 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 155 candidate_status { last_received { term: 154 index: 125 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:29.515228 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 155 candidate_status { last_received { term: 154 index: 125 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:29.515437 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 155 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 155: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:29.516028 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 155 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 155.
I20260317 21:03:29.516260 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 155 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:29.516387 30156 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 155 FOLLOWER]: Leader election won for term 155
I20260317 21:03:29.516496 30156 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 155 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:29.516574 30156 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 154.125, Last appended by leader: 125, Current term: 155, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.517081 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 154 to 155, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 155 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:29.548919 30147 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 155 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:29.549021 30147 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 155 FOLLOWER]: Advancing to term 156
I20260317 21:03:29.550041 30147 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.550225 30147 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 156 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:29.550401 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 156 candidate_status { last_received { term: 154 index: 125 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:29.550515 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 156 candidate_status { last_received { term: 154 index: 125 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:29.550566 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 155 FOLLOWER]: Advancing to term 156
I20260317 21:03:29.551105 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 156 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 156.
I20260317 21:03:29.551323 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 156 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:29.551427 30147 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 156 FOLLOWER]: Leader election won for term 156
I20260317 21:03:29.551481 30147 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 156 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:29.551535 30147 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 154.125, Last appended by leader: 125, Current term: 156, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.552016 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 155 to 156, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 156 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:29.592411 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 156 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 155. Current term is 156. Ops: []
I20260317 21:03:29.592972 30161 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20260317 21:03:29.593143 30156 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 155 LEADER]: Stepping down as leader of term 155
I20260317 21:03:29.593217 30156 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 155 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:29.593264 30156 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 155.126, Last appended by leader: 126, Current term: 155, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.593329 30156 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 155 FOLLOWER]: Advancing to term 156
I20260317 21:03:29.641762 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 156 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 155 index: 126. Preceding OpId from leader: term: 156 index: 126. (term mismatch)
I20260317 21:03:29.641849 29886 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 125
I20260317 21:03:29.641865 29886 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 155.126
I20260317 21:03:29.641878 29886 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 156 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:29.642210 30147 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20260317 21:03:29.643316 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 156 FOLLOWER]: Refusing update from remote peer 0b4ef85c36ee46d8bea7064e0759a98a: Log matching property violated. Preceding OpId in replica: term: 154 index: 125. Preceding OpId from leader: term: 156 index: 126. (index mismatch)
I20260317 21:03:29.643648 30147 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20260317 21:03:29.744719 30160 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 156 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:29.744814 30160 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 156 FOLLOWER]: Advancing to term 157
I20260317 21:03:29.745728 30160 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.745934 30160 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 157 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:29.746235 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 157 candidate_status { last_received { term: 156 index: 126 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:29.746363 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 157 candidate_status { last_received { term: 156 index: 126 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:29.746655 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 157 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:29.746786 30160 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 157 FOLLOWER]: Leader election lost for term 157. Reason: could not achieve majority
I20260317 21:03:29.754762 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 157 FOLLOWER]: Rejecting Update request from peer 0b4ef85c36ee46d8bea7064e0759a98a for earlier term 156. Current term is 157. Ops: []
I20260317 21:03:29.755085 30162 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 156.126, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20260317 21:03:29.755223 30162 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 156 LEADER]: Stepping down as leader of term 156
I20260317 21:03:29.755254 30162 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 156 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:29.755308 30162 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 156.126, Last appended by leader: 126, Current term: 156, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.755405 30162 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 156 FOLLOWER]: Advancing to term 157
I20260317 21:03:29.843421 30156 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 156 FOLLOWER]: Starting leader election (detected failure of leader 0b4ef85c36ee46d8bea7064e0759a98a)
I20260317 21:03:29.843542 30156 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 156 FOLLOWER]: Advancing to term 157
I20260317 21:03:29.844292 30156 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.844483 30156 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 157 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:29.844698 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 157 candidate_status { last_received { term: 156 index: 126 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:29.844837 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 157 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 157: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:29.844710 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 157 candidate_status { last_received { term: 156 index: 126 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:29.846091 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 157 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 157.
I20260317 21:03:29.846323 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 157 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:29.846467 30156 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 157 FOLLOWER]: Leader election won for term 157
I20260317 21:03:29.846552 30156 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 157 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:29.846622 30156 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 156.126, Last appended by leader: 126, Current term: 157, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.847045 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 156 to 157, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 157 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:29.862254 30160 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 157 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:29.862375 30160 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 157 FOLLOWER]: Advancing to term 158
I20260317 21:03:29.863094 30160 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.863279 30160 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 158 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:29.863644 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 158 candidate_status { last_received { term: 156 index: 126 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:29.863763 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 157 FOLLOWER]: Advancing to term 158
I20260317 21:03:29.863675 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 158 candidate_status { last_received { term: 156 index: 126 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:29.864534 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 158 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 158.
I20260317 21:03:29.864846 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 158 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a; no voters: 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:29.864969 30160 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 158 FOLLOWER]: Leader election won for term 158
I20260317 21:03:29.865059 30160 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 158 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:29.865132 30160 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 156.126, Last appended by leader: 126, Current term: 158, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.865727 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 157 to 158, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 158 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:29.896533 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:29.922644 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 158 LEADER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 157. Current term is 158. Ops: []
I20260317 21:03:29.922957 30156 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20260317 21:03:29.923084 30156 consensus_queue.cc:1243] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer 9f1d952e629c4010ae48963a9cdee04a log is divergent from this leader: its last log entry 158.127 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 126
I20260317 21:03:29.923121 30161 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 157 LEADER]: Stepping down as leader of term 157
I20260317 21:03:29.923172 30161 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 157 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:29.923264 30161 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 157.127, Last appended by leader: 127, Current term: 157, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:29.923349 30161 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 157 FOLLOWER]: Advancing to term 158
I20260317 21:03:29.953418 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 158 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 157 index: 127. Preceding OpId from leader: term: 158 index: 127. (term mismatch)
I20260317 21:03:29.953521 29886 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting all ops after (but not including) 126
I20260317 21:03:29.953543 29886 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting uncommitted NO_OP operation due to leader change: 157.127
I20260317 21:03:29.953575 29886 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 158 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:29.953851 30160 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20260317 21:03:29.955070 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 158 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 156 index: 126. Preceding OpId from leader: term: 158 index: 127. (index mismatch)
I20260317 21:03:29.955292 30163 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20260317 21:03:30.151783 30155 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 158 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:30.151897 30155 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 158 FOLLOWER]: Advancing to term 159
I20260317 21:03:30.152665 30155 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.152866 30155 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 159 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:30.153108 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 159 candidate_status { last_received { term: 158 index: 127 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:30.153136 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 159 candidate_status { last_received { term: 158 index: 127 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:30.153548 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 159 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:30.153669 30155 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 159 FOLLOWER]: Leader election lost for term 159. Reason: could not achieve majority
I20260317 21:03:30.155599 30161 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 158 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:30.155671 30161 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 158 FOLLOWER]: Advancing to term 159
I20260317 21:03:30.156435 30161 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.156699 30161 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 159 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:30.156976 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 159 candidate_status { last_received { term: 158 index: 127 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:30.156976 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 159 candidate_status { last_received { term: 158 index: 127 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:30.157142 29961 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 159 FOLLOWER]: Leader election vote request: Denying vote to candidate 24015e78ad6341a9a2bc13754a55b581 in current term 159: Already voted for candidate 0b4ef85c36ee46d8bea7064e0759a98a in this term.
I20260317 21:03:30.157522 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 159 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:30.157693 30161 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 159 FOLLOWER]: Leader election lost for term 159. Reason: could not achieve majority
I20260317 21:03:30.165952 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 159 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 158. Current term is 159. Ops: []
I20260317 21:03:30.166285 30163 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 158.127, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20260317 21:03:30.166447 30154 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 158 LEADER]: Stepping down as leader of term 158
I20260317 21:03:30.166498 30154 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 158 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:30.166567 30154 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 158.127, Last appended by leader: 127, Current term: 158, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.166651 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 158 FOLLOWER]: Advancing to term 159
I20260317 21:03:30.264621 30161 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 159 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:30.264743 30161 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 159 FOLLOWER]: Advancing to term 160
I20260317 21:03:30.265762 30161 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.265959 30161 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 160 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:30.266155 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 160 candidate_status { last_received { term: 158 index: 127 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:30.266215 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 160 candidate_status { last_received { term: 158 index: 127 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:30.266296 29811 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 159 FOLLOWER]: Advancing to term 160
I20260317 21:03:30.266296 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 159 FOLLOWER]: Advancing to term 160
I20260317 21:03:30.266968 29811 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 160 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 160.
I20260317 21:03:30.266976 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 160 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 160.
I20260317 21:03:30.267237 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 160 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters:
I20260317 21:03:30.267369 30161 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 160 FOLLOWER]: Leader election won for term 160
I20260317 21:03:30.267457 30161 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 160 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:30.267514 30161 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 158.127, Last appended by leader: 127, Current term: 160, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.268025 29707 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 158 to 160, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 160 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:30.385586 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 160 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 158 index: 127. Preceding OpId from leader: term: 160 index: 128. (index mismatch)
I20260317 21:03:30.385907 30161 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20260317 21:03:30.386574 29811 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 160 FOLLOWER]: Refusing update from remote peer 24015e78ad6341a9a2bc13754a55b581: Log matching property violated. Preceding OpId in replica: term: 158 index: 127. Preceding OpId from leader: term: 160 index: 128. (index mismatch)
I20260317 21:03:30.386842 30156 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20260317 21:03:30.571491 30154 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 160 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:30.571591 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 160 FOLLOWER]: Advancing to term 161
I20260317 21:03:30.572269 30154 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.572402 30154 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 161 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:30.572846 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 161 candidate_status { last_received { term: 160 index: 128 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:30.572846 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 161 candidate_status { last_received { term: 160 index: 128 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:30.573150 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 161 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581
I20260317 21:03:30.573307 30154 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 161 FOLLOWER]: Leader election lost for term 161. Reason: could not achieve majority
I20260317 21:03:30.586529 29811 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 161 FOLLOWER]: Rejecting Update request from peer 24015e78ad6341a9a2bc13754a55b581 for earlier term 160. Current term is 161. Ops: []
I20260317 21:03:30.586858 30156 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 }, Status: INVALID_TERM, Last received: 160.128, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20260317 21:03:30.587019 30161 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 160 LEADER]: Stepping down as leader of term 160
I20260317 21:03:30.587069 30161 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 160 LEADER]: Becoming Follower/Learner. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:30.587139 30161 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 160.128, Last appended by leader: 128, Current term: 160, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.587239 30161 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 160 FOLLOWER]: Advancing to term 161
I20260317 21:03:30.667335 30155 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 160 FOLLOWER]: Starting leader election (detected failure of leader 24015e78ad6341a9a2bc13754a55b581)
I20260317 21:03:30.667451 30155 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 160 FOLLOWER]: Advancing to term 161
I20260317 21:03:30.668146 30155 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.668378 30155 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 161 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:30.668635 29811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 161 candidate_status { last_received { term: 160 index: 128 } } ignore_live_leader: false dest_uuid: "9f1d952e629c4010ae48963a9cdee04a"
I20260317 21:03:30.668635 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 161 candidate_status { last_received { term: 160 index: 128 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:30.668767 29811 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 161 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 161: Already voted for candidate 9f1d952e629c4010ae48963a9cdee04a in this term.
I20260317 21:03:30.669423 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 161 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4ef85c36ee46d8bea7064e0759a98a in term 161.
I20260317 21:03:30.669616 29926 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 161 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:30.669773 30155 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 161 FOLLOWER]: Leader election won for term 161
I20260317 21:03:30.669931 30155 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 161 LEADER]: Becoming Leader. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:30.670035 30155 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 160.128, Last appended by leader: 128, Current term: 161, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.670503 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a reported cstate change: term changed from 160 to 161, leader changed from 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66) to 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67). New cstate: current_term: 161 leader_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:30.690147 30154 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 161 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:30.690258 30154 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 161 FOLLOWER]: Advancing to term 162
I20260317 21:03:30.690936 30154 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 162 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.691131 30154 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 162 election: Requested vote from peers 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:30.691404 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 162 candidate_status { last_received { term: 160 index: 128 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:30.691474 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "9f1d952e629c4010ae48963a9cdee04a" candidate_term: 162 candidate_status { last_received { term: 160 index: 128 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:30.691586 29886 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 161 FOLLOWER]: Advancing to term 162
I20260317 21:03:30.692160 29886 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 162 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1d952e629c4010ae48963a9cdee04a in term 162.
I20260317 21:03:30.692462 29775 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [CANDIDATE]: Term 162 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a; no voters: 0b4ef85c36ee46d8bea7064e0759a98a
I20260317 21:03:30.692574 30154 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 162 FOLLOWER]: Leader election won for term 162
I20260317 21:03:30.692641 30154 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 162 LEADER]: Becoming Leader. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:30.692691 30154 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 160.128, Last appended by leader: 128, Current term: 162, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.693295 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a reported cstate change: term changed from 161 to 162, leader changed from 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67) to 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65). New cstate: current_term: 162 leader_uuid: "9f1d952e629c4010ae48963a9cdee04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:30.774720 29886 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 162 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 160 index: 128. Preceding OpId from leader: term: 162 index: 129. (index mismatch)
I20260317 21:03:30.775130 30154 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20260317 21:03:30.776293 29961 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 161 LEADER]: Stepping down as leader of term 161
I20260317 21:03:30.776346 29961 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 161 LEADER]: Becoming Follower/Learner. State: Replica: 0b4ef85c36ee46d8bea7064e0759a98a, State: Running, Role: LEADER
I20260317 21:03:30.776400 29961 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 161.129, Last appended by leader: 129, Current term: 161, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.776484 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 161 FOLLOWER]: Advancing to term 162
I20260317 21:03:30.777562 29961 raft_consensus.cc:1275] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 162 FOLLOWER]: Refusing update from remote peer 9f1d952e629c4010ae48963a9cdee04a: Log matching property violated. Preceding OpId in replica: term: 161 index: 129. Preceding OpId from leader: term: 162 index: 129. (term mismatch)
I20260317 21:03:30.777618 29961 pending_rounds.cc:85] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting all ops after (but not including) 128
I20260317 21:03:30.777633 29961 pending_rounds.cc:107] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: Aborting uncommitted NO_OP operation due to leader change: 161.129
I20260317 21:03:30.777647 29961 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 162 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:30.777940 30154 consensus_queue.cc:1048] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20260317 21:03:30.896871 29677 timestamp_advancement-itest.cc:267] GCing logs...
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:267: Failure
Expected: (gcable_size) > (0), actual: 0 vs 0
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:332: Failure
Expected: GCUntilNoWritesInWAL(ts, replica) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260317 21:03:30.897214 29677 tablet_server.cc:178] TabletServer@127.28.251.65:0 shutting down...
I20260317 21:03:30.966187 30161 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 162 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:30.966296 30161 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 162 FOLLOWER]: Advancing to term 163
I20260317 21:03:30.967065 30161 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 163 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.967310 30161 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 163 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:30.967552 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 163 candidate_status { last_received { term: 162 index: 129 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
W20260317 21:03:30.967660 29853 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260317 21:03:30.968259 29853 leader_election.cc:336] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 163 election: RPC error from VoteRequest() call to peer 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260317 21:03:30.968312 29853 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 163 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24015e78ad6341a9a2bc13754a55b581; no voters: 0b4ef85c36ee46d8bea7064e0759a98a, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:30.968402 30161 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 163 FOLLOWER]: Leader election lost for term 163. Reason: could not achieve majority
I20260317 21:03:30.989065 29886 raft_consensus.cc:1240] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 163 FOLLOWER]: Rejecting Update request from peer 9f1d952e629c4010ae48963a9cdee04a for earlier term 162. Current term is 163. Ops: []
I20260317 21:03:30.989487 30166 consensus_queue.cc:1059] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 }, Status: INVALID_TERM, Last received: 162.129, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20260317 21:03:30.989610 30166 raft_consensus.cc:3055] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 162 LEADER]: Stepping down as leader of term 162
I20260317 21:03:30.989629 30166 raft_consensus.cc:740] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 162 LEADER]: Becoming Follower/Learner. State: Replica: 9f1d952e629c4010ae48963a9cdee04a, State: Running, Role: LEADER
I20260317 21:03:30.989710 30166 consensus_queue.cc:260] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 162.129, Last appended by leader: 129, Current term: 162, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:30.989826 30166 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 162 FOLLOWER]: Advancing to term 163
I20260317 21:03:31.002213 30155 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 162 FOLLOWER]: Starting leader election (detected failure of leader 9f1d952e629c4010ae48963a9cdee04a)
I20260317 21:03:31.002388 30155 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 162 FOLLOWER]: Advancing to term 163
I20260317 21:03:31.003060 30155 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 163 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:31.003288 30155 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 163 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66:33653)
I20260317 21:03:31.003575 29886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" candidate_term: 163 candidate_status { last_received { term: 162 index: 129 } } ignore_live_leader: false dest_uuid: "24015e78ad6341a9a2bc13754a55b581"
I20260317 21:03:31.003683 29886 raft_consensus.cc:2393] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b4ef85c36ee46d8bea7064e0759a98a in current term 163: Already voted for candidate 24015e78ad6341a9a2bc13754a55b581 in this term.
W20260317 21:03:31.003961 29928 leader_election.cc:336] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 163 election: RPC error from VoteRequest() call to peer 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260317 21:03:31.004019 29928 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [CANDIDATE]: Term 163 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a; no voters: 24015e78ad6341a9a2bc13754a55b581, 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:31.004088 30155 raft_consensus.cc:2749] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 163 FOLLOWER]: Leader election lost for term 163. Reason: could not achieve majority
I20260317 21:03:31.076833 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:31.076992 29677 tablet_replica.cc:333] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a: stopping tablet replica
I20260317 21:03:31.077088 29677 raft_consensus.cc:2243] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 163 FOLLOWER]: Raft consensus shutting down.
I20260317 21:03:31.077147 29677 raft_consensus.cc:2272] T e2b6dd1d6b8c42a9999eed01df93529f P 9f1d952e629c4010ae48963a9cdee04a [term 163 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:31.083204 30161 raft_consensus.cc:493] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 163 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:31.083294 30161 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 163 FOLLOWER]: Advancing to term 164
I20260317 21:03:31.083982 30161 raft_consensus.cc:515] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 164 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:31.084192 30161 leader_election.cc:290] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 164 election: Requested vote from peers 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093), 0b4ef85c36ee46d8bea7064e0759a98a (127.28.251.67:33237)
I20260317 21:03:31.084440 29961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2b6dd1d6b8c42a9999eed01df93529f" candidate_uuid: "24015e78ad6341a9a2bc13754a55b581" candidate_term: 164 candidate_status { last_received { term: 162 index: 129 } } ignore_live_leader: false dest_uuid: "0b4ef85c36ee46d8bea7064e0759a98a"
I20260317 21:03:31.084524 29961 raft_consensus.cc:3060] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 163 FOLLOWER]: Advancing to term 164
W20260317 21:03:31.084885 29853 leader_election.cc:336] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 164 election: RPC error from VoteRequest() call to peer 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65:41093): Network error: Client connection negotiation failed: client connection to 127.28.251.65:41093: connect: Connection refused (error 111)
I20260317 21:03:31.085407 29961 raft_consensus.cc:2468] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 164 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24015e78ad6341a9a2bc13754a55b581 in term 164.
I20260317 21:03:31.085597 29851 leader_election.cc:304] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [CANDIDATE]: Term 164 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b4ef85c36ee46d8bea7064e0759a98a, 24015e78ad6341a9a2bc13754a55b581; no voters: 9f1d952e629c4010ae48963a9cdee04a
I20260317 21:03:31.085695 30161 raft_consensus.cc:2804] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 164 FOLLOWER]: Leader election won for term 164
I20260317 21:03:31.085780 30161 raft_consensus.cc:697] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 164 LEADER]: Becoming Leader. State: Replica: 24015e78ad6341a9a2bc13754a55b581, State: Running, Role: LEADER
I20260317 21:03:31.085839 30161 consensus_queue.cc:237] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 162.129, Last appended by leader: 129, Current term: 164, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } }
I20260317 21:03:31.086319 29706 catalog_manager.cc:5671] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 reported cstate change: term changed from 162 to 164, leader changed from 9f1d952e629c4010ae48963a9cdee04a (127.28.251.65) to 24015e78ad6341a9a2bc13754a55b581 (127.28.251.66). New cstate: current_term: 164 leader_uuid: "24015e78ad6341a9a2bc13754a55b581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d952e629c4010ae48963a9cdee04a" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4ef85c36ee46d8bea7064e0759a98a" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24015e78ad6341a9a2bc13754a55b581" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:31.090744 29677 tablet_server.cc:195] TabletServer@127.28.251.65:0 shutdown complete.
I20260317 21:03:31.092250 29677 tablet_server.cc:178] TabletServer@127.28.251.66:0 shutting down...
I20260317 21:03:31.107575 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:31.107795 29677 tablet_replica.cc:333] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: stopping tablet replica
I20260317 21:03:31.107864 29677 raft_consensus.cc:2243] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 164 LEADER]: Raft consensus shutting down.
I20260317 21:03:31.107934 29677 pending_rounds.cc:70] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Trying to abort 1 pending ops.
I20260317 21:03:31.107970 29677 pending_rounds.cc:77] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581: Aborting op as it isn't in flight: id { term: 164 index: 130 } timestamp: 7265408659808219136 op_type: NO_OP noop_request { }
I20260317 21:03:31.108059 29677 raft_consensus.cc:2889] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 164 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:03:31.108095 29677 raft_consensus.cc:2272] T e2b6dd1d6b8c42a9999eed01df93529f P 24015e78ad6341a9a2bc13754a55b581 [term 164 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:31.111263 29677 tablet_server.cc:195] TabletServer@127.28.251.66:0 shutdown complete.
I20260317 21:03:31.112368 29677 tablet_server.cc:178] TabletServer@127.28.251.67:0 shutting down...
I20260317 21:03:31.146648 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:31.146806 29677 tablet_replica.cc:333] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a: stopping tablet replica
I20260317 21:03:31.146860 29677 raft_consensus.cc:2243] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 164 FOLLOWER]: Raft consensus shutting down.
I20260317 21:03:31.146934 29677 raft_consensus.cc:2272] T e2b6dd1d6b8c42a9999eed01df93529f P 0b4ef85c36ee46d8bea7064e0759a98a [term 164 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:31.159421 29677 tablet_server.cc:195] TabletServer@127.28.251.67:0 shutdown complete.
I20260317 21:03:31.160881 29677 master.cc:561] Master@127.28.251.126:34775 shutting down...
I20260317 21:03:31.261300 29677 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:03:31.261417 29677 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:31.261444 29677 tablet_replica.cc:333] T 00000000000000000000000000000000 P 5401cbf7a89e418baf84f08087c1c527: stopping tablet replica
I20260317 21:03:31.273952 29677 master.cc:583] Master@127.28.251.126:34775 shutdown complete.
I20260317 21:03:31.276028 29677 test_util.cc:183] -----------------------------------------------
I20260317 21:03:31.276085 29677 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1773781378956650-29677-0
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData (32315 ms)
[ RUN ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap
I20260317 21:03:31.276568 29677 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.251.126:42049
I20260317 21:03:31.276700 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:31.278363 30171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:31.278921 30173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:31.278926 29677 server_base.cc:1060] running on GCE node
W20260317 21:03:31.279037 30170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:31.279206 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:31.279249 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:31.279263 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781411279263 us; error 0 us; skew 500 ppm
I20260317 21:03:31.279856 29677 webserver.cc:492] Webserver started at http://127.28.251.126:45915/ using document root <none> and password file <none>
I20260317 21:03:31.279932 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:31.279965 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:31.280000 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:31.280243 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/master-0-root/instance:
uuid: "60e2225f9d4547a49757c1874335775f"
format_stamp: "Formatted at 2026-03-17 21:03:31 on dist-test-slave-tm0n"
I20260317 21:03:31.281167 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:31.281848 30178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:31.282007 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260317 21:03:31.282065 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/master-0-root
uuid: "60e2225f9d4547a49757c1874335775f"
format_stamp: "Formatted at 2026-03-17 21:03:31 on dist-test-slave-tm0n"
I20260317 21:03:31.282110 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:31.291548 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:31.291766 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:31.294898 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.126:42049
I20260317 21:03:31.297019 30230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.126:42049 every 8 connection(s)
I20260317 21:03:31.297286 30231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:03:31.298429 30231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f: Bootstrap starting.
I20260317 21:03:31.298764 30231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f: Neither blocks nor log segments found. Creating new log.
I20260317 21:03:31.299301 30231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f: No bootstrap required, opened a new log
I20260317 21:03:31.299429 30231 raft_consensus.cc:359] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e2225f9d4547a49757c1874335775f" member_type: VOTER }
I20260317 21:03:31.299474 30231 raft_consensus.cc:385] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:03:31.299487 30231 raft_consensus.cc:740] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e2225f9d4547a49757c1874335775f, State: Initialized, Role: FOLLOWER
I20260317 21:03:31.299526 30231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e2225f9d4547a49757c1874335775f" member_type: VOTER }
I20260317 21:03:31.299549 30231 raft_consensus.cc:399] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 21:03:31.299563 30231 raft_consensus.cc:493] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 21:03:31.299576 30231 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:03:31.300168 30231 raft_consensus.cc:515] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e2225f9d4547a49757c1874335775f" member_type: VOTER }
I20260317 21:03:31.300230 30231 leader_election.cc:304] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60e2225f9d4547a49757c1874335775f; no voters:
I20260317 21:03:31.300359 30231 leader_election.cc:290] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260317 21:03:31.300490 30234 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:03:31.300557 30231 sys_catalog.cc:565] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:03:31.300666 30234 raft_consensus.cc:697] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 1 LEADER]: Becoming Leader. State: Replica: 60e2225f9d4547a49757c1874335775f, State: Running, Role: LEADER
I20260317 21:03:31.300720 30234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e2225f9d4547a49757c1874335775f" member_type: VOTER }
I20260317 21:03:31.300987 30236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60e2225f9d4547a49757c1874335775f. Latest consensus state: current_term: 1 leader_uuid: "60e2225f9d4547a49757c1874335775f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e2225f9d4547a49757c1874335775f" member_type: VOTER } }
I20260317 21:03:31.300957 30235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60e2225f9d4547a49757c1874335775f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e2225f9d4547a49757c1874335775f" member_type: VOTER } }
I20260317 21:03:31.301227 30235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [sys.catalog]: This master's current role is: LEADER
I20260317 21:03:31.301390 30244 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:03:31.301416 30236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [sys.catalog]: This master's current role is: LEADER
I20260317 21:03:31.301610 30244 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:03:31.301994 29677 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260317 21:03:31.302225 30244 catalog_manager.cc:1357] Generated new cluster ID: f73cefc5a3a04ebab279928c9d3590d6
I20260317 21:03:31.302277 30244 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:03:31.312877 30244 catalog_manager.cc:1380] Generated new certificate authority record
I20260317 21:03:31.313489 30244 catalog_manager.cc:1514] Loading token signing keys...
I20260317 21:03:31.317130 30244 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f: Generated new TSK 0
I20260317 21:03:31.317405 30244 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:03:31.334043 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260317 21:03:31.334206 29677 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260317 21:03:31.336340 30253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:31.336393 30252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:31.336357 30255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:31.336701 29677 server_base.cc:1060] running on GCE node
I20260317 21:03:31.336822 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:31.336853 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:31.336877 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781411336877 us; error 0 us; skew 500 ppm
I20260317 21:03:31.337689 29677 webserver.cc:492] Webserver started at http://127.28.251.65:45331/ using document root <none> and password file <none>
I20260317 21:03:31.337783 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:31.337839 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:31.337895 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:31.338187 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-0-root/instance:
uuid: "238010c8cbd0400e9ce7ce7b001d7490"
format_stamp: "Formatted at 2026-03-17 21:03:31 on dist-test-slave-tm0n"
I20260317 21:03:31.339113 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:31.339628 30260 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:31.339926 29677 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:31.340014 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-0-root
uuid: "238010c8cbd0400e9ce7ce7b001d7490"
format_stamp: "Formatted at 2026-03-17 21:03:31 on dist-test-slave-tm0n"
I20260317 21:03:31.340091 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:31.348856 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:31.349121 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:31.349377 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:31.349658 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:03:31.349690 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:31.349713 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:03:31.349726 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:31.352906 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.65:37513
I20260317 21:03:31.352928 30323 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.65:37513 every 8 connection(s)
I20260317 21:03:31.353406 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260317 21:03:31.353531 29677 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260317 21:03:31.354713 30328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:31.354732 30329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:31.354904 29677 server_base.cc:1060] running on GCE node
W20260317 21:03:31.354815 30331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:31.355053 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:31.355108 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:31.355126 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781411355126 us; error 0 us; skew 500 ppm
I20260317 21:03:31.355739 29677 webserver.cc:492] Webserver started at http://127.28.251.66:37103/ using document root <none> and password file <none>
I20260317 21:03:31.355840 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:31.355892 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:31.355959 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:31.356266 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-1-root/instance:
uuid: "d81c47deccb8463da2024e0c84198282"
format_stamp: "Formatted at 2026-03-17 21:03:31 on dist-test-slave-tm0n"
I20260317 21:03:31.357412 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260317 21:03:31.357889 30336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:31.358035 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260317 21:03:31.358088 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-1-root
uuid: "d81c47deccb8463da2024e0c84198282"
format_stamp: "Formatted at 2026-03-17 21:03:31 on dist-test-slave-tm0n"
I20260317 21:03:31.358126 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:31.361299 30324 heartbeater.cc:344] Connected to a master server at 127.28.251.126:42049
I20260317 21:03:31.361397 30324 heartbeater.cc:461] Registering TS with master...
I20260317 21:03:31.361518 30324 heartbeater.cc:507] Master 127.28.251.126:42049 requested a full tablet report, sending...
I20260317 21:03:31.361801 30195 ts_manager.cc:194] Registered new tserver with Master: 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513)
I20260317 21:03:31.362392 30195 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59236
I20260317 21:03:31.369673 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:31.369905 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:31.370096 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:31.370311 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:03:31.370339 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:31.370362 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:03:31.370401 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:31.373461 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.66:38865
I20260317 21:03:31.373522 30399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.66:38865 every 8 connection(s)
I20260317 21:03:31.373890 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260317 21:03:31.374010 29677 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260317 21:03:31.375469 30403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:31.375476 30406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:31.375483 30404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:31.375739 29677 server_base.cc:1060] running on GCE node
I20260317 21:03:31.375900 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:31.375946 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:31.375993 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781411375993 us; error 0 us; skew 500 ppm
I20260317 21:03:31.376747 29677 webserver.cc:492] Webserver started at http://127.28.251.67:39095/ using document root <none> and password file <none>
I20260317 21:03:31.376847 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:31.376905 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:31.376971 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:31.377353 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-2-root/instance:
uuid: "76e99639d71041668f4bed4de6997034"
format_stamp: "Formatted at 2026-03-17 21:03:31 on dist-test-slave-tm0n"
I20260317 21:03:31.377377 30400 heartbeater.cc:344] Connected to a master server at 127.28.251.126:42049
I20260317 21:03:31.377508 30400 heartbeater.cc:461] Registering TS with master...
I20260317 21:03:31.377626 30400 heartbeater.cc:507] Master 127.28.251.126:42049 requested a full tablet report, sending...
I20260317 21:03:31.377849 30195 ts_manager.cc:194] Registered new tserver with Master: d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:31.378237 30195 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59242
I20260317 21:03:31.378470 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:31.379050 30411 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:31.379346 29677 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:31.379420 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-2-root
uuid: "76e99639d71041668f4bed4de6997034"
format_stamp: "Formatted at 2026-03-17 21:03:31 on dist-test-slave-tm0n"
I20260317 21:03:31.379475 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:31.392753 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:31.393069 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:31.393286 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:31.393519 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:03:31.393570 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:31.393612 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:03:31.393639 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:31.396899 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.67:41455
I20260317 21:03:31.396999 30474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.67:41455 every 8 connection(s)
I20260317 21:03:31.400244 30475 heartbeater.cc:344] Connected to a master server at 127.28.251.126:42049
I20260317 21:03:31.400309 30475 heartbeater.cc:461] Registering TS with master...
I20260317 21:03:31.400430 30475 heartbeater.cc:507] Master 127.28.251.126:42049 requested a full tablet report, sending...
I20260317 21:03:31.400730 30195 ts_manager.cc:194] Registered new tserver with Master: 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455)
I20260317 21:03:31.401146 30195 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59244
I20260317 21:03:31.401643 29677 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004509623s
I20260317 21:03:31.413620 30195 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59278:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:03:31.413762 30195 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:03:31.418954 30288 tablet_service.cc:1511] Processing CreateTablet for tablet 74347beb082b4133bae4a16e4d899749 (DEFAULT_TABLE table=test-workload [id=c86ec7a853234d1c9ca167da60351e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:03:31.419055 30364 tablet_service.cc:1511] Processing CreateTablet for tablet 74347beb082b4133bae4a16e4d899749 (DEFAULT_TABLE table=test-workload [id=c86ec7a853234d1c9ca167da60351e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:03:31.419126 30288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74347beb082b4133bae4a16e4d899749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:03:31.419227 30364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74347beb082b4133bae4a16e4d899749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:03:31.420518 30496 tablet_bootstrap.cc:492] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Bootstrap starting.
I20260317 21:03:31.420913 30439 tablet_service.cc:1511] Processing CreateTablet for tablet 74347beb082b4133bae4a16e4d899749 (DEFAULT_TABLE table=test-workload [id=c86ec7a853234d1c9ca167da60351e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:03:31.421036 30439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74347beb082b4133bae4a16e4d899749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:03:31.421129 30496 tablet_bootstrap.cc:654] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Neither blocks nor log segments found. Creating new log.
I20260317 21:03:31.421536 30497 tablet_bootstrap.cc:492] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282: Bootstrap starting.
I20260317 21:03:31.421746 30496 tablet_bootstrap.cc:492] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: No bootstrap required, opened a new log
I20260317 21:03:31.421797 30496 ts_tablet_manager.cc:1403] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:31.421934 30496 raft_consensus.cc:359] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.421990 30496 raft_consensus.cc:385] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:03:31.422004 30496 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238010c8cbd0400e9ce7ce7b001d7490, State: Initialized, Role: FOLLOWER
I20260317 21:03:31.422063 30496 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.422225 30496 ts_tablet_manager.cc:1434] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260317 21:03:31.422279 30324 heartbeater.cc:499] Master 127.28.251.126:42049 was elected leader, sending a full tablet report...
I20260317 21:03:31.423172 30500 tablet_bootstrap.cc:492] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034: Bootstrap starting.
I20260317 21:03:31.423568 30497 tablet_bootstrap.cc:654] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282: Neither blocks nor log segments found. Creating new log.
I20260317 21:03:31.423763 30500 tablet_bootstrap.cc:654] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034: Neither blocks nor log segments found. Creating new log.
I20260317 21:03:31.424206 30497 tablet_bootstrap.cc:492] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282: No bootstrap required, opened a new log
I20260317 21:03:31.424280 30497 ts_tablet_manager.cc:1403] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260317 21:03:31.424335 30500 tablet_bootstrap.cc:492] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034: No bootstrap required, opened a new log
I20260317 21:03:31.424414 30500 ts_tablet_manager.cc:1403] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:31.424443 30497 raft_consensus.cc:359] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.424516 30497 raft_consensus.cc:385] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:03:31.424547 30497 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d81c47deccb8463da2024e0c84198282, State: Initialized, Role: FOLLOWER
I20260317 21:03:31.424577 30500 raft_consensus.cc:359] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.424620 30497 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.424649 30500 raft_consensus.cc:385] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:03:31.424700 30500 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e99639d71041668f4bed4de6997034, State: Initialized, Role: FOLLOWER
I20260317 21:03:31.424775 30500 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.424803 30497 ts_tablet_manager.cc:1434] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260317 21:03:31.424841 30400 heartbeater.cc:499] Master 127.28.251.126:42049 was elected leader, sending a full tablet report...
I20260317 21:03:31.424975 30475 heartbeater.cc:499] Master 127.28.251.126:42049 was elected leader, sending a full tablet report...
I20260317 21:03:31.425027 30500 ts_tablet_manager.cc:1434] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:31.426251 30499 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:03:31.426301 30499 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.426432 30499 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:31.428970 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034" is_pre_election: true
I20260317 21:03:31.429065 30449 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 238010c8cbd0400e9ce7ce7b001d7490 in term 0.
I20260317 21:03:31.429365 30263 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 238010c8cbd0400e9ce7ce7b001d7490, 76e99639d71041668f4bed4de6997034; no voters:
I20260317 21:03:31.429491 30499 raft_consensus.cc:2804] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:03:31.429538 30499 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:31.429562 30499 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:03:31.429560 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282" is_pre_election: true
I20260317 21:03:31.429625 30373 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 238010c8cbd0400e9ce7ce7b001d7490 in term 0.
I20260317 21:03:31.430243 30499 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.430375 30499 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 1 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:31.430536 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:31.430599 30449 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:03:31.430568 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:31.430699 30373 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:03:31.431396 30373 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 238010c8cbd0400e9ce7ce7b001d7490 in term 1.
I20260317 21:03:31.431404 30449 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 238010c8cbd0400e9ce7ce7b001d7490 in term 1.
I20260317 21:03:31.431605 30263 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 238010c8cbd0400e9ce7ce7b001d7490, 76e99639d71041668f4bed4de6997034; no voters:
I20260317 21:03:31.431720 30499 raft_consensus.cc:2804] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:03:31.431895 30499 raft_consensus.cc:697] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 1 LEADER]: Becoming Leader. State: Replica: 238010c8cbd0400e9ce7ce7b001d7490, State: Running, Role: LEADER
I20260317 21:03:31.431962 30499 consensus_queue.cc:237] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.432550 30195 catalog_manager.cc:5671] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 reported cstate change: term changed from 0 to 1, leader changed from <none> to 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65). New cstate: current_term: 1 leader_uuid: "238010c8cbd0400e9ce7ce7b001d7490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:31.441409 30373 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 1 FOLLOWER]: Refusing update from remote peer 238010c8cbd0400e9ce7ce7b001d7490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:03:31.441416 30449 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 1 FOLLOWER]: Refusing update from remote peer 238010c8cbd0400e9ce7ce7b001d7490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:03:31.441696 30499 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:03:31.441849 30506 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:03:31.443361 30513 mvcc.cc:204] Tried to move back new op lower bound from 7265408661263609856 to 7265408661225832448. Current Snapshot: MvccSnapshot[applied={T|T < 7265408661263609856 or (T in {7265408661263609856})}]
I20260317 21:03:31.677546 30504 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 238010c8cbd0400e9ce7ce7b001d7490)
I20260317 21:03:31.677654 30504 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:03:31.678399 30504 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.678629 30504 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 2 election: Requested vote from peers 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:31.679574 30449 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 2 FOLLOWER]: Rejecting Update request from peer 238010c8cbd0400e9ce7ce7b001d7490 for earlier term 1. Current term is 2. Ops: []
I20260317 21:03:31.679816 30531 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 }, Status: INVALID_TERM, Last received: 1.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260317 21:03:31.681602 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:31.682000 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:31.682260 30414 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76e99639d71041668f4bed4de6997034; no voters: 238010c8cbd0400e9ce7ce7b001d7490, d81c47deccb8463da2024e0c84198282
I20260317 21:03:31.682376 30504 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260317 21:03:31.745095 30518 raft_consensus.cc:3055] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 1 LEADER]: Stepping down as leader of term 1
I20260317 21:03:31.745275 30518 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 238010c8cbd0400e9ce7ce7b001d7490, State: Running, Role: LEADER
I20260317 21:03:31.745376 30518 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.745537 30518 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:03:31.792835 30504 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:31.792941 30504 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 2 FOLLOWER]: Advancing to term 3
I20260317 21:03:31.793694 30504 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.793895 30504 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 3 election: Requested vote from peers 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:31.794121 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 3 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:31.794119 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 3 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:31.794281 30298 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 2 FOLLOWER]: Advancing to term 3
I20260317 21:03:31.794890 30298 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76e99639d71041668f4bed4de6997034 in term 3.
I20260317 21:03:31.795117 30413 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 238010c8cbd0400e9ce7ce7b001d7490, 76e99639d71041668f4bed4de6997034; no voters: d81c47deccb8463da2024e0c84198282
I20260317 21:03:31.795238 30504 raft_consensus.cc:2804] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 21:03:31.795351 30504 raft_consensus.cc:697] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 3 LEADER]: Becoming Leader. State: Replica: 76e99639d71041668f4bed4de6997034, State: Running, Role: LEADER
I20260317 21:03:31.795398 30504 consensus_queue.cc:237] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.795914 30194 catalog_manager.cc:5671] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 reported cstate change: term changed from 1 to 3, leader changed from 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65) to 76e99639d71041668f4bed4de6997034 (127.28.251.67). New cstate: current_term: 3 leader_uuid: "76e99639d71041668f4bed4de6997034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:31.823614 30503 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 238010c8cbd0400e9ce7ce7b001d7490)
I20260317 21:03:31.823737 30503 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:03:31.824558 30503 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:31.824801 30503 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 2 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513)
I20260317 21:03:31.827564 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:31.827665 30298 raft_consensus.cc:2368] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d81c47deccb8463da2024e0c84198282 for earlier term 2. Current term is 3.
I20260317 21:03:31.827736 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:31.827853 30338 leader_election.cc:400] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 2 election: Vote denied by peer 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513) with higher term. Message: Invalid argument: T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d81c47deccb8463da2024e0c84198282 for earlier term 2. Current term is 3.
I20260317 21:03:31.827929 30338 leader_election.cc:403] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260317 21:03:31.827999 30503 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 2 FOLLOWER]: Advancing to term 3
I20260317 21:03:31.828666 30503 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513) with higher term. Message: Invalid argument: T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d81c47deccb8463da2024e0c84198282 for earlier term 2. Current term is 3.
I20260317 21:03:31.885519 30298 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 3 FOLLOWER]: Refusing update from remote peer 76e99639d71041668f4bed4de6997034: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20260317 21:03:31.885890 30504 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260317 21:03:31.898705 30373 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 3 FOLLOWER]: Refusing update from remote peer 76e99639d71041668f4bed4de6997034: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20260317 21:03:31.899016 30504 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260317 21:03:32.000459 30503 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 76e99639d71041668f4bed4de6997034)
I20260317 21:03:32.000571 30503 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 3 FOLLOWER]: Advancing to term 4
I20260317 21:03:32.001384 30503 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.001577 30503 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 4 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513)
I20260317 21:03:32.001760 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:32.001806 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:32.002048 30338 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d81c47deccb8463da2024e0c84198282; no voters: 238010c8cbd0400e9ce7ce7b001d7490, 76e99639d71041668f4bed4de6997034
I20260317 21:03:32.002164 30503 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260317 21:03:32.013036 30373 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 4 FOLLOWER]: Rejecting Update request from peer 76e99639d71041668f4bed4de6997034 for earlier term 3. Current term is 4. Ops: []
I20260317 21:03:32.013624 30536 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 }, Status: INVALID_TERM, Last received: 3.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260317 21:03:32.094571 30373 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 4 FOLLOWER]: Rejecting Update request from peer 76e99639d71041668f4bed4de6997034 for earlier term 3. Current term is 4. Ops: []
I20260317 21:03:32.094950 30536 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 }, Status: INVALID_TERM, Last received: 3.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260317 21:03:32.112650 30504 raft_consensus.cc:3055] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 21:03:32.112793 30504 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 76e99639d71041668f4bed4de6997034, State: Running, Role: LEADER
I20260317 21:03:32.112876 30504 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.17, Last appended by leader: 17, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.113026 30504 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 3 FOLLOWER]: Advancing to term 4
I20260317 21:03:32.142081 30503 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:32.142191 30503 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 4 FOLLOWER]: Advancing to term 5
I20260317 21:03:32.142943 30503 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.143126 30503 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 5 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513)
I20260317 21:03:32.143369 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 5 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:32.143383 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 5 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:32.143479 30449 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 4 FOLLOWER]: Advancing to term 5
I20260317 21:03:32.144101 30449 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d81c47deccb8463da2024e0c84198282 in term 5.
I20260317 21:03:32.144335 30339 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76e99639d71041668f4bed4de6997034, d81c47deccb8463da2024e0c84198282; no voters: 238010c8cbd0400e9ce7ce7b001d7490
I20260317 21:03:32.144485 30503 raft_consensus.cc:2804] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 21:03:32.144593 30503 raft_consensus.cc:697] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 5 LEADER]: Becoming Leader. State: Replica: d81c47deccb8463da2024e0c84198282, State: Running, Role: LEADER
I20260317 21:03:32.144640 30503 consensus_queue.cc:237] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.17, Last appended by leader: 17, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.145396 30194 catalog_manager.cc:5671] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 reported cstate change: term changed from 3 to 5, leader changed from 76e99639d71041668f4bed4de6997034 (127.28.251.67) to d81c47deccb8463da2024e0c84198282 (127.28.251.66). New cstate: current_term: 5 leader_uuid: "d81c47deccb8463da2024e0c84198282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:32.152983 30518 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 76e99639d71041668f4bed4de6997034)
I20260317 21:03:32.153087 30518 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 3 FOLLOWER]: Advancing to term 4
I20260317 21:03:32.153855 30518 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.154026 30518 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 4 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:32.154275 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:32.154338 30449 raft_consensus.cc:2368] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 238010c8cbd0400e9ce7ce7b001d7490 for earlier term 4. Current term is 5.
I20260317 21:03:32.154335 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:32.154647 30263 leader_election.cc:400] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 4 election: Vote denied by peer 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455) with higher term. Message: Invalid argument: T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 238010c8cbd0400e9ce7ce7b001d7490 for earlier term 4. Current term is 5.
I20260317 21:03:32.154716 30263 leader_election.cc:403] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260317 21:03:32.154799 30518 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 4 FOLLOWER]: Advancing to term 5
I20260317 21:03:32.155433 30518 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455) with higher term. Message: Invalid argument: T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 238010c8cbd0400e9ce7ce7b001d7490 for earlier term 4. Current term is 5.
I20260317 21:03:32.235862 30449 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 5 FOLLOWER]: Refusing update from remote peer d81c47deccb8463da2024e0c84198282: Log matching property violated. Preceding OpId in replica: term: 3 index: 17. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
I20260317 21:03:32.236340 30503 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260317 21:03:32.246965 30298 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 5 FOLLOWER]: Refusing update from remote peer d81c47deccb8463da2024e0c84198282: Log matching property violated. Preceding OpId in replica: term: 3 index: 17. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
I20260317 21:03:32.247331 30538 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260317 21:03:32.338953 30504 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d81c47deccb8463da2024e0c84198282)
I20260317 21:03:32.339161 30504 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 5 FOLLOWER]: Advancing to term 6
I20260317 21:03:32.339831 30504 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.340051 30504 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 6 election: Requested vote from peers 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:32.340256 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:32.340267 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:32.340575 30414 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76e99639d71041668f4bed4de6997034; no voters: 238010c8cbd0400e9ce7ce7b001d7490, d81c47deccb8463da2024e0c84198282
I20260317 21:03:32.340739 30504 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260317 21:03:32.344431 30449 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 6 FOLLOWER]: Rejecting Update request from peer d81c47deccb8463da2024e0c84198282 for earlier term 5. Current term is 6. Ops: []
I20260317 21:03:32.344805 30503 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 }, Status: INVALID_TERM, Last received: 5.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260317 21:03:32.397428 30538 raft_consensus.cc:3055] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 5 LEADER]: Stepping down as leader of term 5
I20260317 21:03:32.397574 30538 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d81c47deccb8463da2024e0c84198282, State: Running, Role: LEADER
I20260317 21:03:32.397677 30538 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.397830 30538 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 5 FOLLOWER]: Advancing to term 6
I20260317 21:03:32.434206 30518 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d81c47deccb8463da2024e0c84198282)
I20260317 21:03:32.434309 30518 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 5 FOLLOWER]: Advancing to term 6
I20260317 21:03:32.434967 30518 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.435151 30518 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 6 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:32.435410 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:32.435467 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:32.435532 30449 raft_consensus.cc:2393] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 238010c8cbd0400e9ce7ce7b001d7490 in current term 6: Already voted for candidate 76e99639d71041668f4bed4de6997034 in this term.
I20260317 21:03:32.436224 30373 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 238010c8cbd0400e9ce7ce7b001d7490 in term 6.
I20260317 21:03:32.436460 30263 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 238010c8cbd0400e9ce7ce7b001d7490, d81c47deccb8463da2024e0c84198282; no voters: 76e99639d71041668f4bed4de6997034
I20260317 21:03:32.436576 30518 raft_consensus.cc:2804] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 6 FOLLOWER]: Leader election won for term 6
I20260317 21:03:32.436707 30518 raft_consensus.cc:697] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 6 LEADER]: Becoming Leader. State: Replica: 238010c8cbd0400e9ce7ce7b001d7490, State: Running, Role: LEADER
I20260317 21:03:32.436782 30518 consensus_queue.cc:237] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.437649 30194 catalog_manager.cc:5671] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 reported cstate change: term changed from 5 to 6, leader changed from d81c47deccb8463da2024e0c84198282 (127.28.251.66) to 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65). New cstate: current_term: 6 leader_uuid: "238010c8cbd0400e9ce7ce7b001d7490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:32.446156 30504 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:32.446271 30504 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 6 FOLLOWER]: Advancing to term 7
I20260317 21:03:32.446959 30504 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.447249 30504 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 7 election: Requested vote from peers 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:32.447417 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 7 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:32.447403 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 7 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:32.447567 30373 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 6 FOLLOWER]: Advancing to term 7
I20260317 21:03:32.448132 30373 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76e99639d71041668f4bed4de6997034 in term 7.
I20260317 21:03:32.448354 30414 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76e99639d71041668f4bed4de6997034, d81c47deccb8463da2024e0c84198282; no voters: 238010c8cbd0400e9ce7ce7b001d7490
I20260317 21:03:32.448458 30504 raft_consensus.cc:2804] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 7 FOLLOWER]: Leader election won for term 7
I20260317 21:03:32.448529 30504 raft_consensus.cc:697] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 7 LEADER]: Becoming Leader. State: Replica: 76e99639d71041668f4bed4de6997034, State: Running, Role: LEADER
I20260317 21:03:32.448582 30504 consensus_queue.cc:237] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.525621 30373 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 7 FOLLOWER]: Rejecting Update request from peer 238010c8cbd0400e9ce7ce7b001d7490 for earlier term 6. Current term is 7. Ops: []
I20260317 21:03:32.526028 30518 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260317 21:03:32.529824 30195 catalog_manager.cc:5671] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 reported cstate change: term changed from 6 to 7, leader changed from 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65) to 76e99639d71041668f4bed4de6997034 (127.28.251.67). New cstate: current_term: 7 leader_uuid: "76e99639d71041668f4bed4de6997034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:32.532676 30298 raft_consensus.cc:3055] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 6 LEADER]: Stepping down as leader of term 6
I20260317 21:03:32.532781 30298 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 238010c8cbd0400e9ce7ce7b001d7490, State: Running, Role: LEADER
I20260317 21:03:32.532843 30298 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 6.19, Last appended by leader: 19, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.532961 30298 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 6 FOLLOWER]: Advancing to term 7
I20260317 21:03:32.533865 30298 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 7 FOLLOWER]: Refusing update from remote peer 76e99639d71041668f4bed4de6997034: Log matching property violated. Preceding OpId in replica: term: 6 index: 19. Preceding OpId from leader: term: 7 index: 19. (term mismatch)
I20260317 21:03:32.534121 30298 pending_rounds.cc:85] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Aborting all ops after (but not including) 18
I20260317 21:03:32.534176 30298 pending_rounds.cc:107] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Aborting uncommitted NO_OP operation due to leader change: 6.19
I20260317 21:03:32.534209 30298 raft_consensus.cc:2889] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:32.534569 30504 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260317 21:03:32.548666 30373 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 7 FOLLOWER]: Refusing update from remote peer 76e99639d71041668f4bed4de6997034: Log matching property violated. Preceding OpId in replica: term: 5 index: 18. Preceding OpId from leader: term: 7 index: 19. (index mismatch)
I20260317 21:03:32.549000 30504 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260317 21:03:32.825511 30538 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 76e99639d71041668f4bed4de6997034)
I20260317 21:03:32.825665 30538 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 7 FOLLOWER]: Advancing to term 8
I20260317 21:03:32.826373 30538 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.826594 30538 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 8 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513)
I20260317 21:03:32.826795 30297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 8 candidate_status { last_received { term: 7 index: 19 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:32.826907 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 8 candidate_status { last_received { term: 7 index: 19 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:32.827117 30339 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d81c47deccb8463da2024e0c84198282; no voters: 238010c8cbd0400e9ce7ce7b001d7490, 76e99639d71041668f4bed4de6997034
I20260317 21:03:32.827256 30538 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260317 21:03:32.834061 30373 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 8 FOLLOWER]: Rejecting Update request from peer 76e99639d71041668f4bed4de6997034 for earlier term 7. Current term is 8. Ops: []
I20260317 21:03:32.834393 30504 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 }, Status: INVALID_TERM, Last received: 7.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260317 21:03:32.901657 30533 raft_consensus.cc:3055] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 7 LEADER]: Stepping down as leader of term 7
I20260317 21:03:32.901755 30533 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 76e99639d71041668f4bed4de6997034, State: Running, Role: LEADER
I20260317 21:03:32.901825 30533 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 7.19, Last appended by leader: 19, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.902012 30533 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 7 FOLLOWER]: Advancing to term 8
I20260317 21:03:32.927549 30518 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 76e99639d71041668f4bed4de6997034)
I20260317 21:03:32.927659 30518 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 7 FOLLOWER]: Advancing to term 8
I20260317 21:03:32.928336 30518 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.928467 30518 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 8 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:32.928757 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 8 candidate_status { last_received { term: 7 index: 19 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:32.928792 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 8 candidate_status { last_received { term: 7 index: 19 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:32.928867 30373 raft_consensus.cc:2393] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 238010c8cbd0400e9ce7ce7b001d7490 in current term 8: Already voted for candidate d81c47deccb8463da2024e0c84198282 in this term.
I20260317 21:03:32.929489 30449 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 238010c8cbd0400e9ce7ce7b001d7490 in term 8.
I20260317 21:03:32.929721 30263 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 238010c8cbd0400e9ce7ce7b001d7490, 76e99639d71041668f4bed4de6997034; no voters: d81c47deccb8463da2024e0c84198282
I20260317 21:03:32.929865 30518 raft_consensus.cc:2804] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 8 FOLLOWER]: Leader election won for term 8
I20260317 21:03:32.929960 30518 raft_consensus.cc:697] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 8 LEADER]: Becoming Leader. State: Replica: 238010c8cbd0400e9ce7ce7b001d7490, State: Running, Role: LEADER
I20260317 21:03:32.930030 30518 consensus_queue.cc:237] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 7.19, Last appended by leader: 19, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.930505 30194 catalog_manager.cc:5671] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 reported cstate change: term changed from 7 to 8, leader changed from 76e99639d71041668f4bed4de6997034 (127.28.251.67) to 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65). New cstate: current_term: 8 leader_uuid: "238010c8cbd0400e9ce7ce7b001d7490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:32.969904 30538 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:32.970043 30538 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 8 FOLLOWER]: Advancing to term 9
I20260317 21:03:32.970816 30538 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.971045 30538 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 9 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513)
I20260317 21:03:32.971259 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 9 candidate_status { last_received { term: 7 index: 19 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:32.971321 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 9 candidate_status { last_received { term: 7 index: 19 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:32.971383 30449 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 8 FOLLOWER]: Advancing to term 9
I20260317 21:03:32.971985 30449 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d81c47deccb8463da2024e0c84198282 in term 9.
I20260317 21:03:32.972213 30339 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76e99639d71041668f4bed4de6997034, d81c47deccb8463da2024e0c84198282; no voters: 238010c8cbd0400e9ce7ce7b001d7490
I20260317 21:03:32.972342 30538 raft_consensus.cc:2804] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 9 FOLLOWER]: Leader election won for term 9
I20260317 21:03:32.972486 30538 raft_consensus.cc:697] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 9 LEADER]: Becoming Leader. State: Replica: d81c47deccb8463da2024e0c84198282, State: Running, Role: LEADER
I20260317 21:03:32.972548 30538 consensus_queue.cc:237] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 7.19, Last appended by leader: 19, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:32.973057 30194 catalog_manager.cc:5671] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 reported cstate change: term changed from 8 to 9, leader changed from 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65) to d81c47deccb8463da2024e0c84198282 (127.28.251.66). New cstate: current_term: 9 leader_uuid: "d81c47deccb8463da2024e0c84198282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:33.031112 30449 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 9 FOLLOWER]: Rejecting Update request from peer 238010c8cbd0400e9ce7ce7b001d7490 for earlier term 8. Current term is 9. Ops: []
I20260317 21:03:33.031555 30518 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260317 21:03:33.051417 30373 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 9 LEADER]: Rejecting Update request from peer 238010c8cbd0400e9ce7ce7b001d7490 for earlier term 8. Current term is 9. Ops: []
I20260317 21:03:33.051817 30539 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260317 21:03:33.051919 30539 consensus_queue.cc:1243] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Peer d81c47deccb8463da2024e0c84198282 log is divergent from this leader: its last log entry 9.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260317 21:03:33.086378 30449 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 9 FOLLOWER]: Refusing update from remote peer d81c47deccb8463da2024e0c84198282: Log matching property violated. Preceding OpId in replica: term: 7 index: 19. Preceding OpId from leader: term: 9 index: 20. (index mismatch)
I20260317 21:03:33.086727 30538 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260317 21:03:33.088815 30518 raft_consensus.cc:3055] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 8 LEADER]: Stepping down as leader of term 8
I20260317 21:03:33.088900 30518 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 238010c8cbd0400e9ce7ce7b001d7490, State: Running, Role: LEADER
I20260317 21:03:33.088981 30518 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 8.20, Last appended by leader: 20, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.089093 30518 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 8 FOLLOWER]: Advancing to term 9
I20260317 21:03:33.095538 30298 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 9 FOLLOWER]: Refusing update from remote peer d81c47deccb8463da2024e0c84198282: Log matching property violated. Preceding OpId in replica: term: 8 index: 20. Preceding OpId from leader: term: 9 index: 20. (term mismatch)
I20260317 21:03:33.095618 30298 pending_rounds.cc:85] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Aborting all ops after (but not including) 19
I20260317 21:03:33.095633 30298 pending_rounds.cc:107] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Aborting uncommitted NO_OP operation due to leader change: 8.20
I20260317 21:03:33.095647 30298 raft_consensus.cc:2889] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:03:33.095906 30538 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260317 21:03:33.371905 30533 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d81c47deccb8463da2024e0c84198282)
I20260317 21:03:33.372038 30533 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 9 FOLLOWER]: Advancing to term 10
I20260317 21:03:33.372735 30533 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.372949 30533 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 10 election: Requested vote from peers 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:33.373075 30518 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d81c47deccb8463da2024e0c84198282)
I20260317 21:03:33.373142 30518 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 9 FOLLOWER]: Advancing to term 10
I20260317 21:03:33.373231 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 10 candidate_status { last_received { term: 9 index: 20 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:33.373250 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 10 candidate_status { last_received { term: 9 index: 20 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:33.373811 30518 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.373917 30298 raft_consensus.cc:2393] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 76e99639d71041668f4bed4de6997034 in current term 10: Already voted for candidate 238010c8cbd0400e9ce7ce7b001d7490 in this term.
I20260317 21:03:33.373948 30518 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 10 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:33.374114 30413 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76e99639d71041668f4bed4de6997034; no voters: 238010c8cbd0400e9ce7ce7b001d7490, d81c47deccb8463da2024e0c84198282
I20260317 21:03:33.374125 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 10 candidate_status { last_received { term: 9 index: 20 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:33.374208 30533 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260317 21:03:33.374169 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 10 candidate_status { last_received { term: 9 index: 20 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:33.374408 30449 raft_consensus.cc:2393] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 238010c8cbd0400e9ce7ce7b001d7490 in current term 10: Already voted for candidate 76e99639d71041668f4bed4de6997034 in this term.
I20260317 21:03:33.374548 30263 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 238010c8cbd0400e9ce7ce7b001d7490; no voters: 76e99639d71041668f4bed4de6997034, d81c47deccb8463da2024e0c84198282
I20260317 21:03:33.374637 30518 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260317 21:03:33.394152 30449 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 10 FOLLOWER]: Rejecting Update request from peer d81c47deccb8463da2024e0c84198282 for earlier term 9. Current term is 10. Ops: []
I20260317 21:03:33.394520 30544 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 }, Status: INVALID_TERM, Last received: 9.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260317 21:03:33.397074 30298 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 10 FOLLOWER]: Rejecting Update request from peer d81c47deccb8463da2024e0c84198282 for earlier term 9. Current term is 10. Ops: []
I20260317 21:03:33.397461 30538 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 }, Status: INVALID_TERM, Last received: 9.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260317 21:03:33.459798 30544 raft_consensus.cc:3055] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 9 LEADER]: Stepping down as leader of term 9
I20260317 21:03:33.459906 30544 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d81c47deccb8463da2024e0c84198282, State: Running, Role: LEADER
I20260317 21:03:33.460019 30544 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 9.20, Last appended by leader: 20, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.460136 30544 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 9 FOLLOWER]: Advancing to term 10
I20260317 21:03:33.498065 30518 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:33.498186 30518 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 10 FOLLOWER]: Advancing to term 11
I20260317 21:03:33.498875 30518 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.499043 30518 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 11 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:33.499293 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 11 candidate_status { last_received { term: 9 index: 20 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:33.499372 30373 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 10 FOLLOWER]: Advancing to term 11
I20260317 21:03:33.499316 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "238010c8cbd0400e9ce7ce7b001d7490" candidate_term: 11 candidate_status { last_received { term: 9 index: 20 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:33.499449 30449 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 10 FOLLOWER]: Advancing to term 11
I20260317 21:03:33.499979 30373 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 238010c8cbd0400e9ce7ce7b001d7490 in term 11.
I20260317 21:03:33.500099 30449 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 238010c8cbd0400e9ce7ce7b001d7490 in term 11.
I20260317 21:03:33.500208 30263 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 238010c8cbd0400e9ce7ce7b001d7490, d81c47deccb8463da2024e0c84198282; no voters:
I20260317 21:03:33.500324 30518 raft_consensus.cc:2804] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 11 FOLLOWER]: Leader election won for term 11
I20260317 21:03:33.500401 30518 raft_consensus.cc:697] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 11 LEADER]: Becoming Leader. State: Replica: 238010c8cbd0400e9ce7ce7b001d7490, State: Running, Role: LEADER
I20260317 21:03:33.500458 30518 consensus_queue.cc:237] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 9.20, Last appended by leader: 20, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.501427 30194 catalog_manager.cc:5671] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 reported cstate change: term changed from 9 to 11, leader changed from d81c47deccb8463da2024e0c84198282 (127.28.251.66) to 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65). New cstate: current_term: 11 leader_uuid: "238010c8cbd0400e9ce7ce7b001d7490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:33.507525 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:33.508829 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:33.511138 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:33.515415 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:33.523684 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:33.539996 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:33.572328 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:33.593628 30449 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 11 FOLLOWER]: Refusing update from remote peer 238010c8cbd0400e9ce7ce7b001d7490: Log matching property violated. Preceding OpId in replica: term: 9 index: 20. Preceding OpId from leader: term: 11 index: 21. (index mismatch)
I20260317 21:03:33.593945 30518 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260317 21:03:33.594935 30373 raft_consensus.cc:1275] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 11 FOLLOWER]: Refusing update from remote peer 238010c8cbd0400e9ce7ce7b001d7490: Log matching property violated. Preceding OpId in replica: term: 9 index: 20. Preceding OpId from leader: term: 11 index: 21. (index mismatch)
I20260317 21:03:33.595171 30548 consensus_queue.cc:1048] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260317 21:03:33.636610 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:33.636787 29677 log_reader.cc:385] T 74347beb082b4133bae4a16e4d899749: removed 1 log segments from log reader
I20260317 21:03:33.636848 29677 log.cc:1079] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Deleting log segment in path: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-0-root/wals/74347beb082b4133bae4a16e4d899749/wal-000000001 (ops 1-16)
I20260317 21:03:33.637385 29677 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20260317 21:03:33.637476 29677 master.cc:561] Master@127.28.251.126:42049 shutting down...
I20260317 21:03:33.695427 30533 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 238010c8cbd0400e9ce7ce7b001d7490)
I20260317 21:03:33.695538 30533 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 11 FOLLOWER]: Advancing to term 12
I20260317 21:03:33.696255 30533 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.696439 30533 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 12 election: Requested vote from peers 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:33.696556 30540 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 238010c8cbd0400e9ce7ce7b001d7490)
I20260317 21:03:33.696602 30540 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 11 FOLLOWER]: Advancing to term 12
I20260317 21:03:33.696699 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:33.696734 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:33.697481 30540 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.697582 30373 raft_consensus.cc:2393] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 76e99639d71041668f4bed4de6997034 in current term 12: Already voted for candidate d81c47deccb8463da2024e0c84198282 in this term.
I20260317 21:03:33.697634 30540 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 12 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513)
I20260317 21:03:33.697760 30414 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76e99639d71041668f4bed4de6997034; no voters: 238010c8cbd0400e9ce7ce7b001d7490, d81c47deccb8463da2024e0c84198282
I20260317 21:03:33.697800 30449 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "76e99639d71041668f4bed4de6997034"
I20260317 21:03:33.697811 30298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "d81c47deccb8463da2024e0c84198282" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "238010c8cbd0400e9ce7ce7b001d7490"
I20260317 21:03:33.697863 30533 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260317 21:03:33.697916 30449 raft_consensus.cc:2393] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate d81c47deccb8463da2024e0c84198282 in current term 12: Already voted for candidate 76e99639d71041668f4bed4de6997034 in this term.
I20260317 21:03:33.698060 30339 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d81c47deccb8463da2024e0c84198282; no voters: 238010c8cbd0400e9ce7ce7b001d7490, 76e99639d71041668f4bed4de6997034
I20260317 21:03:33.698267 30540 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260317 21:03:33.700873 30449 raft_consensus.cc:1240] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 12 FOLLOWER]: Rejecting Update request from peer 238010c8cbd0400e9ce7ce7b001d7490 for earlier term 11. Current term is 12. Ops: []
I20260317 21:03:33.701144 30548 consensus_queue.cc:1059] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 }, Status: INVALID_TERM, Last received: 11.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260317 21:03:33.701299 30548 raft_consensus.cc:3055] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 11 LEADER]: Stepping down as leader of term 11
I20260317 21:03:33.701332 30548 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 238010c8cbd0400e9ce7ce7b001d7490, State: Running, Role: LEADER
I20260317 21:03:33.701442 30548 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 11.21, Last appended by leader: 21, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.701548 30548 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 11 FOLLOWER]: Advancing to term 12
I20260317 21:03:33.793862 29677 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:03:33.794024 29677 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:33.794059 29677 tablet_replica.cc:333] T 00000000000000000000000000000000 P 60e2225f9d4547a49757c1874335775f: stopping tablet replica
I20260317 21:03:33.806450 29677 master.cc:583] Master@127.28.251.126:42049 shutdown complete.
I20260317 21:03:33.807871 29677 tablet_server.cc:178] TabletServer@127.28.251.65:0 shutting down...
I20260317 21:03:33.824433 30533 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:33.824585 30533 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 12 FOLLOWER]: Advancing to term 13
I20260317 21:03:33.825387 30533 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.825606 30533 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 13 election: Requested vote from peers 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
I20260317 21:03:33.825896 30373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74347beb082b4133bae4a16e4d899749" candidate_uuid: "76e99639d71041668f4bed4de6997034" candidate_term: 13 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "d81c47deccb8463da2024e0c84198282"
I20260317 21:03:33.825977 30373 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 12 FOLLOWER]: Advancing to term 13
W20260317 21:03:33.826248 30413 leader_election.cc:336] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 238010c8cbd0400e9ce7ce7b001d7490 (127.28.251.65:37513): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260317 21:03:33.826673 30373 raft_consensus.cc:2468] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76e99639d71041668f4bed4de6997034 in term 13.
I20260317 21:03:33.826895 30414 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76e99639d71041668f4bed4de6997034, d81c47deccb8463da2024e0c84198282; no voters: 238010c8cbd0400e9ce7ce7b001d7490
I20260317 21:03:33.827014 30533 raft_consensus.cc:2804] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 13 FOLLOWER]: Leader election won for term 13
I20260317 21:03:33.827116 30533 raft_consensus.cc:697] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 13 LEADER]: Becoming Leader. State: Replica: 76e99639d71041668f4bed4de6997034, State: Running, Role: LEADER
I20260317 21:03:33.827159 30533 consensus_queue.cc:237] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 11.21, Last appended by leader: 21, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
W20260317 21:03:33.827893 30475 heartbeater.cc:646] Failed to heartbeat to 127.28.251.126:42049 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.251.126:42049: connect: Connection refused (error 111)
I20260317 21:03:33.854859 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:33.855084 29677 tablet_replica.cc:333] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: stopping tablet replica
I20260317 21:03:33.855139 29677 raft_consensus.cc:2243] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 12 FOLLOWER]: Raft consensus shutting down.
I20260317 21:03:33.855204 29677 raft_consensus.cc:2272] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 12 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:33.867565 29677 tablet_server.cc:195] TabletServer@127.28.251.65:0 shutdown complete.
I20260317 21:03:33.868800 29677 tablet_server.cc:178] TabletServer@127.28.251.66:0 shutting down...
I20260317 21:03:33.875330 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:33.875530 29677 tablet_replica.cc:333] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282: stopping tablet replica
I20260317 21:03:33.875586 29677 raft_consensus.cc:2243] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 13 FOLLOWER]: Raft consensus shutting down.
I20260317 21:03:33.875641 29677 raft_consensus.cc:2272] T 74347beb082b4133bae4a16e4d899749 P d81c47deccb8463da2024e0c84198282 [term 13 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:33.877534 29677 tablet_server.cc:195] TabletServer@127.28.251.66:0 shutdown complete.
I20260317 21:03:33.879160 29677 tablet_server.cc:178] TabletServer@127.28.251.67:0 shutting down...
I20260317 21:03:33.898926 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:33.899170 29677 tablet_replica.cc:333] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034: stopping tablet replica
I20260317 21:03:33.899255 29677 raft_consensus.cc:2243] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 13 LEADER]: Raft consensus shutting down.
I20260317 21:03:33.899312 29677 pending_rounds.cc:70] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034: Trying to abort 1 pending ops.
I20260317 21:03:33.899333 29677 pending_rounds.cc:77] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034: Aborting op as it isn't in flight: id { term: 13 index: 22 } timestamp: 7265408671036702720 op_type: NO_OP noop_request { }
I20260317 21:03:33.899385 29677 raft_consensus.cc:2889] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 13 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:03:33.899411 29677 raft_consensus.cc:2272] T 74347beb082b4133bae4a16e4d899749 P 76e99639d71041668f4bed4de6997034 [term 13 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:33.901335 29677 tablet_server.cc:195] TabletServer@127.28.251.67:0 shutdown complete.
I20260317 21:03:33.902140 29677 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20260317 21:03:33.902277 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260317 21:03:33.902415 29677 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260317 21:03:33.903728 30556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:33.903743 30553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:33.903923 29677 server_base.cc:1060] running on GCE node
W20260317 21:03:33.904067 30554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:33.904232 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:33.904279 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:33.904294 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781413904294 us; error 0 us; skew 500 ppm
I20260317 21:03:33.904863 29677 webserver.cc:492] Webserver started at http://127.28.251.65:45331/ using document root <none> and password file <none>
I20260317 21:03:33.904960 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:33.904999 29677 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260317 21:03:33.905807 29677 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:33.907362 30563 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:33.907507 29677 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:33.907560 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-0-root
uuid: "238010c8cbd0400e9ce7ce7b001d7490"
format_stamp: "Formatted at 2026-03-17 21:03:31 on dist-test-slave-tm0n"
I20260317 21:03:33.907604 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1773781378956650-29677-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5295
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:33.912808 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:33.912997 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:33.913133 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:33.913529 30570 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260317 21:03:33.914343 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260317 21:03:33.914398 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:33.914436 29677 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260317 21:03:33.914898 29677 ts_tablet_manager.cc:616] Registered 1 tablets
I20260317 21:03:33.914928 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:33.914964 30570 tablet_bootstrap.cc:492] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Bootstrap starting.
I20260317 21:03:33.916381 30570 tablet_bootstrap.cc:492] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=2 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 21:03:33.916653 30570 tablet_bootstrap.cc:492] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Bootstrap complete.
I20260317 21:03:33.916769 30570 ts_tablet_manager.cc:1403] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260317 21:03:33.916913 30570 raft_consensus.cc:359] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 12 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.916989 30570 raft_consensus.cc:740] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238010c8cbd0400e9ce7ce7b001d7490, State: Initialized, Role: FOLLOWER
I20260317 21:03:33.917081 30570 consensus_queue.cc:260] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 11.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:33.917276 30570 ts_tablet_manager.cc:1434] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260317 21:03:33.918352 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.65:37513
I20260317 21:03:33.920706 30629 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.65:37513 every 8 connection(s)
W20260317 21:03:33.921756 30630 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260317 21:03:33.928414 29677 tablet_server.cc:178] TabletServer@127.28.251.65:37513 shutting down...
I20260317 21:03:34.001497 30610 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:34.001600 30610 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 12 FOLLOWER]: Advancing to term 13
I20260317 21:03:34.002482 30610 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:34.002710 30610 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 13 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
W20260317 21:03:34.003362 30566 leader_election.cc:336] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455): Network error: Client connection negotiation failed: client connection to 127.28.251.67:41455: connect: Connection refused (error 111)
W20260317 21:03:34.003441 30566 leader_election.cc:336] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865): Network error: Client connection negotiation failed: client connection to 127.28.251.66:38865: connect: Connection refused (error 111)
I20260317 21:03:34.003491 30566 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 238010c8cbd0400e9ce7ce7b001d7490; no voters: 76e99639d71041668f4bed4de6997034, d81c47deccb8463da2024e0c84198282
I20260317 21:03:34.003588 30610 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260317 21:03:34.125486 30610 raft_consensus.cc:493] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:34.125635 30610 raft_consensus.cc:3060] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 13 FOLLOWER]: Advancing to term 14
I20260317 21:03:34.126359 30610 raft_consensus.cc:515] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e99639d71041668f4bed4de6997034" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 41455 } } peers { permanent_uuid: "238010c8cbd0400e9ce7ce7b001d7490" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 37513 } } peers { permanent_uuid: "d81c47deccb8463da2024e0c84198282" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 38865 } }
I20260317 21:03:34.126549 30610 leader_election.cc:290] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 14 election: Requested vote from peers 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455), d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865)
W20260317 21:03:34.127265 30566 leader_election.cc:336] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 76e99639d71041668f4bed4de6997034 (127.28.251.67:41455): Network error: Client connection negotiation failed: client connection to 127.28.251.67:41455: connect: Connection refused (error 111)
W20260317 21:03:34.127344 30566 leader_election.cc:336] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer d81c47deccb8463da2024e0c84198282 (127.28.251.66:38865): Network error: Client connection negotiation failed: client connection to 127.28.251.66:38865: connect: Connection refused (error 111)
I20260317 21:03:34.127362 30566 leader_election.cc:304] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 238010c8cbd0400e9ce7ce7b001d7490; no voters: 76e99639d71041668f4bed4de6997034, d81c47deccb8463da2024e0c84198282
I20260317 21:03:34.127477 30610 raft_consensus.cc:2749] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260317 21:03:34.171218 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:34.171456 29677 tablet_replica.cc:333] stopping tablet replica
I20260317 21:03:34.171540 29677 raft_consensus.cc:2243] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 14 FOLLOWER]: Raft consensus shutting down.
I20260317 21:03:34.171613 29677 raft_consensus.cc:2272] T 74347beb082b4133bae4a16e4d899749 P 238010c8cbd0400e9ce7ce7b001d7490 [term 14 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:34.183588 29677 tablet_server.cc:195] TabletServer@127.28.251.65:37513 shutdown complete.
[ OK ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap (2912 ms)
[ RUN ] TimestampAdvancementITest.Kudu2463Test
I20260317 21:03:34.188823 29677 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.251.126:46457
I20260317 21:03:34.188972 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:34.190416 30637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:34.190550 29677 server_base.cc:1060] running on GCE node
W20260317 21:03:34.190582 30638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:34.190802 30640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:34.190966 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:34.191021 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:34.191044 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781414191043 us; error 0 us; skew 500 ppm
I20260317 21:03:34.191717 29677 webserver.cc:492] Webserver started at http://127.28.251.126:38223/ using document root <none> and password file <none>
I20260317 21:03:34.191831 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:34.191897 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:34.191982 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:34.192333 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/master-0-root/instance:
uuid: "9827f9c4c6ce47b59236413621303c0f"
format_stamp: "Formatted at 2026-03-17 21:03:34 on dist-test-slave-tm0n"
I20260317 21:03:34.193343 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260317 21:03:34.193874 30645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.194144 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260317 21:03:34.194204 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/master-0-root
uuid: "9827f9c4c6ce47b59236413621303c0f"
format_stamp: "Formatted at 2026-03-17 21:03:34 on dist-test-slave-tm0n"
I20260317 21:03:34.194274 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:34.206106 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:34.206382 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:34.209153 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.126:46457
I20260317 21:03:34.209882 30697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.126:46457 every 8 connection(s)
I20260317 21:03:34.209998 30698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:03:34.211017 30698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f: Bootstrap starting.
I20260317 21:03:34.211323 30698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f: Neither blocks nor log segments found. Creating new log.
I20260317 21:03:34.211840 30698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f: No bootstrap required, opened a new log
I20260317 21:03:34.211977 30698 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9827f9c4c6ce47b59236413621303c0f" member_type: VOTER }
I20260317 21:03:34.212036 30698 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:03:34.212051 30698 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9827f9c4c6ce47b59236413621303c0f, State: Initialized, Role: FOLLOWER
I20260317 21:03:34.212180 30698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9827f9c4c6ce47b59236413621303c0f" member_type: VOTER }
I20260317 21:03:34.212230 30698 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 21:03:34.212245 30698 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 21:03:34.212260 30698 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:03:34.212767 30698 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9827f9c4c6ce47b59236413621303c0f" member_type: VOTER }
I20260317 21:03:34.212828 30698 leader_election.cc:304] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9827f9c4c6ce47b59236413621303c0f; no voters:
I20260317 21:03:34.212955 30698 leader_election.cc:290] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260317 21:03:34.213013 30701 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:03:34.213151 30698 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:03:34.213282 30701 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 1 LEADER]: Becoming Leader. State: Replica: 9827f9c4c6ce47b59236413621303c0f, State: Running, Role: LEADER
I20260317 21:03:34.213346 30701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9827f9c4c6ce47b59236413621303c0f" member_type: VOTER }
I20260317 21:03:34.213558 30702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9827f9c4c6ce47b59236413621303c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9827f9c4c6ce47b59236413621303c0f" member_type: VOTER } }
I20260317 21:03:34.213670 30702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [sys.catalog]: This master's current role is: LEADER
I20260317 21:03:34.213569 30703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9827f9c4c6ce47b59236413621303c0f. Latest consensus state: current_term: 1 leader_uuid: "9827f9c4c6ce47b59236413621303c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9827f9c4c6ce47b59236413621303c0f" member_type: VOTER } }
I20260317 21:03:34.213811 30703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [sys.catalog]: This master's current role is: LEADER
I20260317 21:03:34.214003 30707 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:03:34.214149 30707 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:03:34.214771 29677 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260317 21:03:34.214772 30707 catalog_manager.cc:1357] Generated new cluster ID: 295b8c54330845e8af326044cc70dade
I20260317 21:03:34.214849 30707 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:03:34.236258 30707 catalog_manager.cc:1380] Generated new certificate authority record
I20260317 21:03:34.236739 30707 catalog_manager.cc:1514] Loading token signing keys...
I20260317 21:03:34.241586 30707 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f: Generated new TSK 0
I20260317 21:03:34.241712 30707 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:03:34.246692 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:34.248131 30719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:34.248149 30722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:34.248278 30720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:34.248420 29677 server_base.cc:1060] running on GCE node
I20260317 21:03:34.248548 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:34.248574 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:34.248587 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781414248588 us; error 0 us; skew 500 ppm
I20260317 21:03:34.249114 29677 webserver.cc:492] Webserver started at http://127.28.251.65:37375/ using document root <none> and password file <none>
I20260317 21:03:34.249257 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:34.249300 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:34.249349 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:34.249598 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-0-root/instance:
uuid: "364faf0bbe6c4665b365c26abc0824c4"
format_stamp: "Formatted at 2026-03-17 21:03:34 on dist-test-slave-tm0n"
I20260317 21:03:34.250483 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260317 21:03:34.250979 30727 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.251117 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.251174 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-0-root
uuid: "364faf0bbe6c4665b365c26abc0824c4"
format_stamp: "Formatted at 2026-03-17 21:03:34 on dist-test-slave-tm0n"
I20260317 21:03:34.251238 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:34.258841 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:34.259074 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:34.259245 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:34.259456 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:03:34.259502 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.259544 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:03:34.259572 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.262568 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.65:46151
I20260317 21:03:34.262594 30790 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.65:46151 every 8 connection(s)
I20260317 21:03:34.262962 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:34.264371 30795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:34.264398 30796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:34.264680 30798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:34.264709 29677 server_base.cc:1060] running on GCE node
I20260317 21:03:34.264945 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:34.264994 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:34.265039 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781414265039 us; error 0 us; skew 500 ppm
I20260317 21:03:34.265790 29677 webserver.cc:492] Webserver started at http://127.28.251.66:38553/ using document root <none> and password file <none>
I20260317 21:03:34.265885 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:34.265951 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:34.266013 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:34.266316 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-1-root/instance:
uuid: "fee0cb2d84a6473c8038690c01b873c9"
format_stamp: "Formatted at 2026-03-17 21:03:34 on dist-test-slave-tm0n"
I20260317 21:03:34.266716 30791 heartbeater.cc:344] Connected to a master server at 127.28.251.126:46457
I20260317 21:03:34.266780 30791 heartbeater.cc:461] Registering TS with master...
I20260317 21:03:34.266898 30791 heartbeater.cc:507] Master 127.28.251.126:46457 requested a full tablet report, sending...
I20260317 21:03:34.267155 30662 ts_manager.cc:194] Registered new tserver with Master: 364faf0bbe6c4665b365c26abc0824c4 (127.28.251.65:46151)
I20260317 21:03:34.267683 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260317 21:03:34.267910 30662 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50774
I20260317 21:03:34.268281 30803 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.268405 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260317 21:03:34.268452 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-1-root
uuid: "fee0cb2d84a6473c8038690c01b873c9"
format_stamp: "Formatted at 2026-03-17 21:03:34 on dist-test-slave-tm0n"
I20260317 21:03:34.268502 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:34.278728 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:34.279031 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:34.279232 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:34.279458 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:03:34.279488 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.279513 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:03:34.279526 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.282634 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.66:42085
I20260317 21:03:34.282677 30866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.66:42085 every 8 connection(s)
I20260317 21:03:34.283058 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:34.284744 30871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:34.284751 30870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:34.284859 30873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:34.285329 29677 server_base.cc:1060] running on GCE node
I20260317 21:03:34.285506 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:34.285557 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:34.285579 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781414285579 us; error 0 us; skew 500 ppm
I20260317 21:03:34.286253 29677 webserver.cc:492] Webserver started at http://127.28.251.67:46413/ using document root <none> and password file <none>
I20260317 21:03:34.286388 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:34.286505 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:34.286579 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:34.286818 30867 heartbeater.cc:344] Connected to a master server at 127.28.251.126:46457
I20260317 21:03:34.286881 30867 heartbeater.cc:461] Registering TS with master...
I20260317 21:03:34.286958 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-2-root/instance:
uuid: "15476ec7a2f24a4abbc60a26d9e30e04"
format_stamp: "Formatted at 2026-03-17 21:03:34 on dist-test-slave-tm0n"
I20260317 21:03:34.286996 30867 heartbeater.cc:507] Master 127.28.251.126:46457 requested a full tablet report, sending...
I20260317 21:03:34.287250 30662 ts_manager.cc:194] Registered new tserver with Master: fee0cb2d84a6473c8038690c01b873c9 (127.28.251.66:42085)
I20260317 21:03:34.287722 30662 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50776
I20260317 21:03:34.288034 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260317 21:03:34.288628 30878 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.288874 29677 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:34.288926 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-2-root
uuid: "15476ec7a2f24a4abbc60a26d9e30e04"
format_stamp: "Formatted at 2026-03-17 21:03:34 on dist-test-slave-tm0n"
I20260317 21:03:34.288962 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:34.294422 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:34.294718 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:34.294900 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:34.295107 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:03:34.295266 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.295337 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:03:34.295358 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.298906 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.67:45017
I20260317 21:03:34.298979 30941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.67:45017 every 8 connection(s)
I20260317 21:03:34.302425 30942 heartbeater.cc:344] Connected to a master server at 127.28.251.126:46457
I20260317 21:03:34.302488 30942 heartbeater.cc:461] Registering TS with master...
I20260317 21:03:34.302609 30942 heartbeater.cc:507] Master 127.28.251.126:46457 requested a full tablet report, sending...
I20260317 21:03:34.302865 30662 ts_manager.cc:194] Registered new tserver with Master: 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67:45017)
I20260317 21:03:34.303205 30662 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50790
I20260317 21:03:34.303507 29677 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004366827s
I20260317 21:03:34.317366 30662 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50816:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:03:34.317546 30662 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:03:34.323381 30831 tablet_service.cc:1511] Processing CreateTablet for tablet ca99df8523cd423b9247c879f4eedd90 (DEFAULT_TABLE table=test-workload [id=536bf3ee38324ce5b5a769cde562e9dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:03:34.323556 30831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca99df8523cd423b9247c879f4eedd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:03:34.323625 30906 tablet_service.cc:1511] Processing CreateTablet for tablet ca99df8523cd423b9247c879f4eedd90 (DEFAULT_TABLE table=test-workload [id=536bf3ee38324ce5b5a769cde562e9dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:03:34.323712 30906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca99df8523cd423b9247c879f4eedd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:03:34.324110 30755 tablet_service.cc:1511] Processing CreateTablet for tablet ca99df8523cd423b9247c879f4eedd90 (DEFAULT_TABLE table=test-workload [id=536bf3ee38324ce5b5a769cde562e9dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:03:34.324252 30755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca99df8523cd423b9247c879f4eedd90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:03:34.325404 30963 tablet_bootstrap.cc:492] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9: Bootstrap starting.
I20260317 21:03:34.325536 30964 tablet_bootstrap.cc:492] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04: Bootstrap starting.
I20260317 21:03:34.325878 30965 tablet_bootstrap.cc:492] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: Bootstrap starting.
I20260317 21:03:34.325954 30963 tablet_bootstrap.cc:654] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9: Neither blocks nor log segments found. Creating new log.
I20260317 21:03:34.326025 30964 tablet_bootstrap.cc:654] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04: Neither blocks nor log segments found. Creating new log.
I20260317 21:03:34.326366 30965 tablet_bootstrap.cc:654] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: Neither blocks nor log segments found. Creating new log.
I20260317 21:03:34.326602 30963 tablet_bootstrap.cc:492] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9: No bootstrap required, opened a new log
I20260317 21:03:34.326627 30964 tablet_bootstrap.cc:492] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04: No bootstrap required, opened a new log
I20260317 21:03:34.326684 30963 ts_tablet_manager.cc:1403] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:34.326911 30964 ts_tablet_manager.cc:1403] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:34.326894 30965 tablet_bootstrap.cc:492] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: No bootstrap required, opened a new log
I20260317 21:03:34.326982 30965 ts_tablet_manager.cc:1403] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:34.326895 30963 raft_consensus.cc:359] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.327085 30963 raft_consensus.cc:385] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:03:34.327072 30964 raft_consensus.cc:359] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.327128 30965 raft_consensus.cc:359] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.327168 30964 raft_consensus.cc:385] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:03:34.327140 30963 raft_consensus.cc:740] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee0cb2d84a6473c8038690c01b873c9, State: Initialized, Role: FOLLOWER
I20260317 21:03:34.327212 30964 raft_consensus.cc:740] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15476ec7a2f24a4abbc60a26d9e30e04, State: Initialized, Role: FOLLOWER
I20260317 21:03:34.327248 30963 consensus_queue.cc:260] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.327275 30964 consensus_queue.cc:260] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.327433 30963 ts_tablet_manager.cc:1434] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:34.327433 30964 ts_tablet_manager.cc:1434] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.327484 30867 heartbeater.cc:499] Master 127.28.251.126:46457 was elected leader, sending a full tablet report...
I20260317 21:03:34.327484 30942 heartbeater.cc:499] Master 127.28.251.126:46457 was elected leader, sending a full tablet report...
I20260317 21:03:34.327189 30965 raft_consensus.cc:385] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:03:34.327613 30965 raft_consensus.cc:740] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364faf0bbe6c4665b365c26abc0824c4, State: Initialized, Role: FOLLOWER
I20260317 21:03:34.327661 30965 consensus_queue.cc:260] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.327831 30965 ts_tablet_manager.cc:1434] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:34.327853 30791 heartbeater.cc:499] Master 127.28.251.126:46457 was elected leader, sending a full tablet report...
I20260317 21:03:34.335789 30969 raft_consensus.cc:493] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:03:34.335907 30969 raft_consensus.cc:515] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.336117 30969 leader_election.cc:290] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67:45017), 364faf0bbe6c4665b365c26abc0824c4 (127.28.251.65:46151)
I20260317 21:03:34.338671 30765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca99df8523cd423b9247c879f4eedd90" candidate_uuid: "fee0cb2d84a6473c8038690c01b873c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364faf0bbe6c4665b365c26abc0824c4" is_pre_election: true
I20260317 21:03:34.338766 30765 raft_consensus.cc:2468] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee0cb2d84a6473c8038690c01b873c9 in term 0.
I20260317 21:03:34.338953 30806 leader_election.cc:304] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364faf0bbe6c4665b365c26abc0824c4, fee0cb2d84a6473c8038690c01b873c9; no voters:
I20260317 21:03:34.338954 30916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca99df8523cd423b9247c879f4eedd90" candidate_uuid: "fee0cb2d84a6473c8038690c01b873c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" is_pre_election: true
I20260317 21:03:34.339090 30916 raft_consensus.cc:2468] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee0cb2d84a6473c8038690c01b873c9 in term 0.
I20260317 21:03:34.339192 30969 raft_consensus.cc:2804] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:03:34.339238 30969 raft_consensus.cc:493] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:03:34.339262 30969 raft_consensus.cc:3060] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:03:34.339892 30969 raft_consensus.cc:515] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.340039 30969 leader_election.cc:290] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [CANDIDATE]: Term 1 election: Requested vote from peers 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67:45017), 364faf0bbe6c4665b365c26abc0824c4 (127.28.251.65:46151)
I20260317 21:03:34.340185 30916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca99df8523cd423b9247c879f4eedd90" candidate_uuid: "fee0cb2d84a6473c8038690c01b873c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15476ec7a2f24a4abbc60a26d9e30e04"
I20260317 21:03:34.340247 30765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca99df8523cd423b9247c879f4eedd90" candidate_uuid: "fee0cb2d84a6473c8038690c01b873c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364faf0bbe6c4665b365c26abc0824c4"
I20260317 21:03:34.340255 30916 raft_consensus.cc:3060] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:03:34.340366 30765 raft_consensus.cc:3060] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:03:34.340968 30916 raft_consensus.cc:2468] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee0cb2d84a6473c8038690c01b873c9 in term 1.
I20260317 21:03:34.341089 30765 raft_consensus.cc:2468] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee0cb2d84a6473c8038690c01b873c9 in term 1.
I20260317 21:03:34.341156 30805 leader_election.cc:304] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15476ec7a2f24a4abbc60a26d9e30e04, fee0cb2d84a6473c8038690c01b873c9; no voters:
I20260317 21:03:34.341313 30969 raft_consensus.cc:2804] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:03:34.341482 30969 raft_consensus.cc:697] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Becoming Leader. State: Replica: fee0cb2d84a6473c8038690c01b873c9, State: Running, Role: LEADER
I20260317 21:03:34.341557 30969 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.342206 30661 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fee0cb2d84a6473c8038690c01b873c9 (127.28.251.66). New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:34.353564 30916 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260317 21:03:34.353672 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:03:34.353916 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:03:34.354012 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:03:34.375603 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.376112 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20260317 21:03:34.376148 30916 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20260317 21:03:34.376317 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260317 21:03:34.376420 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260317 21:03:34.377038 30981 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index -1 to 19, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.377415 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index -1 to 19, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.378515 30916 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Committing config change with OpId 1.19: config changed from index -1 to 19, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.378679 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.378518 30662 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 reported cstate change: config changed from index -1 to 19, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.379320 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20260317 21:03:34.379540 30973 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260317 21:03:34.379698 30659 catalog_manager.cc:5349] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca99df8523cd423b9247c879f4eedd90 with cas_config_opid_index 19: no extra replica candidate found for tablet ca99df8523cd423b9247c879f4eedd90 (table test-workload [id=536bf3ee38324ce5b5a769cde562e9dc]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20260317 21:03:34.380019 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20260317 21:03:34.380371 30973 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260317 21:03:34.380765 30969 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.380888 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.381006 30915 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.381935 30662 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 19 to 20, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:34.382019 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.382421 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20260317 21:03:34.382422 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20260317 21:03:34.382714 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260317 21:03:34.382797 30973 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260317 21:03:34.383437 30969 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.383585 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.384493 30661 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 20 to 21, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:34.384552 30915 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.384783 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.385462 30916 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20260317 21:03:34.385562 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20260317 21:03:34.385651 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260317 21:03:34.385766 30973 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260317 21:03:34.386289 30973 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.386389 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.386468 30916 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.387496 30661 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 reported cstate change: config changed from index 21 to 22, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.388514 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.388861 30916 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20260317 21:03:34.388942 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20260317 21:03:34.389042 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260317 21:03:34.389127 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260317 21:03:34.389663 30981 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.389782 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.390568 30916 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.390653 30661 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 reported cstate change: config changed from index 22 to 23, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.390790 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.391180 30764 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20260317 21:03:34.391399 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260317 21:03:34.391930 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20260317 21:03:34.392023 30981 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.392171 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260317 21:03:34.392208 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.392499 30916 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.393040 30659 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 reported cstate change: config changed from index 23 to 24, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.393162 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.393662 30916 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20260317 21:03:34.393882 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260317 21:03:34.393918 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20260317 21:03:34.394073 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260317 21:03:34.394760 30973 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.394960 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.395987 30916 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.396065 30662 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 24 to 25, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:34.396121 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.396833 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20260317 21:03:34.396979 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260317 21:03:34.397002 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20260317 21:03:34.397271 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260317 21:03:34.397830 30969 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.398101 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.398447 30916 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.398945 30662 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 25 to 26, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:34.399091 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.399474 30764 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20260317 21:03:34.399842 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20260317 21:03:34.399875 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260317 21:03:34.400032 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260317 21:03:34.400579 30981 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.400688 30764 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.400954 30915 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.401783 30662 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 26 to 27, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:34.402012 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.402472 30764 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20260317 21:03:34.402544 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20260317 21:03:34.402680 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260317 21:03:34.402797 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260317 21:03:34.403510 30981 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.403640 30915 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.403640 30764 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.404663 30661 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 reported cstate change: config changed from index 27 to 28, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.404752 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.405109 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20260317 21:03:34.405112 30764 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20260317 21:03:34.405395 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260317 21:03:34.405493 30973 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260317 21:03:34.405963 30973 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.406066 30764 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.406878 30659 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 28 to 29, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:34.406885 30915 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.407025 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.407418 30764 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20260317 21:03:34.407578 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20260317 21:03:34.408136 30973 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.408450 30916 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20260317 21:03:34.408650 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20260317 21:03:34.408978 30916 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.409089 30764 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.409293 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.409135 30662 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 29 to 30, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: UNKNOWN } } }
I20260317 21:03:34.410421 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20260317 21:03:34.410606 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20260317 21:03:34.411564 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20260317 21:03:34.411675 30646 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca99df8523cd423b9247c879f4eedd90 with cas_config_opid_index 19: aborting the task: latest config opid_index 30; task opid_index 19
I20260317 21:03:34.411602 30969 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.411814 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20260317 21:03:34.411940 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.412089 30916 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.412891 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.413101 30662 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 reported cstate change: config changed from index 30 to 31, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.413412 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20260317 21:03:34.413463 30916 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20260317 21:03:34.413642 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20260317 21:03:34.413739 30973 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20260317 21:03:34.414309 30973 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.414420 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.415252 30916 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.415285 30659 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 31 to 32, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:34.415391 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.415940 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20260317 21:03:34.416077 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20260317 21:03:34.416311 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20260317 21:03:34.416392 30973 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20260317 21:03:34.416916 30969 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.417039 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.417129 30915 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.417933 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.417969 30661 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 32 to 33, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:34.418278 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20260317 21:03:34.418417 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20260317 21:03:34.418553 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20260317 21:03:34.418705 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20260317 21:03:34.419265 30973 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.419369 30915 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.419795 30648 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca99df8523cd423b9247c879f4eedd90 with cas_config_opid_index 21: aborting the task: latest config opid_index 33; task opid_index 21
I20260317 21:03:34.420106 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.420244 30659 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 33 to 34, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:34.420284 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.420922 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20260317 21:03:34.421159 30764 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20260317 21:03:34.421252 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20260317 21:03:34.421468 30647 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca99df8523cd423b9247c879f4eedd90 with cas_config_opid_index 23: aborting the task: latest config opid_index 34; task opid_index 23
I20260317 21:03:34.421627 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20260317 21:03:34.422226 30969 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.422405 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.423239 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.423239 30659 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 reported cstate change: config changed from index 34 to 35, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.423462 30915 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.423888 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20260317 21:03:34.424063 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20260317 21:03:34.424734 30973 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.424786 30916 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20260317 21:03:34.424839 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.425110 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20260317 21:03:34.425432 30916 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.425802 30661 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 reported cstate change: config changed from index 35 to 36, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.425875 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.426474 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20260317 21:03:34.426608 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20260317 21:03:34.426755 30973 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20260317 21:03:34.426823 30969 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20260317 21:03:34.427269 30973 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index 36 to 37, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.427423 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index 36 to 37, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.427443 30915 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Committing config change with OpId 1.37: config changed from index 36 to 37, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.428292 30659 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 reported cstate change: config changed from index 36 to 37, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: NON_VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.428825 30841 consensus_queue.cc:237] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.429329 30765 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch)
I20260317 21:03:34.429329 30915 raft_consensus.cc:1275] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 LEARNER]: Refusing update from remote peer fee0cb2d84a6473c8038690c01b873c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch)
I20260317 21:03:34.429566 30998 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20260317 21:03:34.429665 30981 consensus_queue.cc:1048] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20260317 21:03:34.430179 30973 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index 37 to 38, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.430287 30915 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index 37 to 38, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.430287 30765 raft_consensus.cc:2955] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index 37 to 38, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } } }
I20260317 21:03:34.430994 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:34.431149 30662 catalog_manager.cc:5671] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 reported cstate change: config changed from index 37 to 38, 15476ec7a2f24a4abbc60a26d9e30e04 (127.28.251.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fee0cb2d84a6473c8038690c01b873c9" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:34.432190 29677 timestamp_advancement-itest.cc:267] GCing logs...
I20260317 21:03:34.432257 29677 log_reader.cc:385] T ca99df8523cd423b9247c879f4eedd90: removed 1 log segments from log reader
I20260317 21:03:34.432287 29677 log.cc:1079] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: Deleting log segment in path: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-0-root/wals/ca99df8523cd423b9247c879f4eedd90/wal-000000001 (ops 1-18)
I20260317 21:03:34.432905 29677 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20260317 21:03:34.432981 29677 master.cc:561] Master@127.28.251.126:46457 shutting down...
I20260317 21:03:34.449870 30646 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca99df8523cd423b9247c879f4eedd90 with cas_config_opid_index 29: aborting the task: latest config opid_index 38; task opid_index 29
I20260317 21:03:34.451004 30648 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca99df8523cd423b9247c879f4eedd90 with cas_config_opid_index 25: aborting the task: latest config opid_index 38; task opid_index 25
I20260317 21:03:34.465304 30648 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca99df8523cd423b9247c879f4eedd90 with cas_config_opid_index 27: aborting the task: latest config opid_index 38; task opid_index 27
I20260317 21:03:34.483361 30647 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca99df8523cd423b9247c879f4eedd90 with cas_config_opid_index 33: aborting the task: latest config opid_index 38; task opid_index 33
I20260317 21:03:34.492595 30648 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca99df8523cd423b9247c879f4eedd90 with cas_config_opid_index 37: aborting the task: latest config opid_index 38; task opid_index 37
I20260317 21:03:34.494000 29677 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:03:34.494136 29677 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:34.494287 29677 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9827f9c4c6ce47b59236413621303c0f: stopping tablet replica
I20260317 21:03:34.497046 29677 master.cc:583] Master@127.28.251.126:46457 shutdown complete.
I20260317 21:03:34.498409 29677 tablet_server.cc:178] TabletServer@127.28.251.65:0 shutting down...
I20260317 21:03:34.513489 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:34.513724 29677 tablet_replica.cc:333] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: stopping tablet replica
I20260317 21:03:34.513787 29677 raft_consensus.cc:2243] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:03:34.513955 29677 raft_consensus.cc:2272] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:34.526402 29677 tablet_server.cc:195] TabletServer@127.28.251.65:0 shutdown complete.
I20260317 21:03:34.527668 29677 tablet_server.cc:178] TabletServer@127.28.251.66:0 shutting down...
I20260317 21:03:34.533552 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:34.533846 29677 tablet_replica.cc:333] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9: stopping tablet replica
I20260317 21:03:34.533903 29677 raft_consensus.cc:2243] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:03:34.534000 29677 raft_consensus.cc:2272] T ca99df8523cd423b9247c879f4eedd90 P fee0cb2d84a6473c8038690c01b873c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:34.547042 29677 tablet_server.cc:195] TabletServer@127.28.251.66:0 shutdown complete.
I20260317 21:03:34.548245 29677 tablet_server.cc:178] TabletServer@127.28.251.67:0 shutting down...
I20260317 21:03:34.799990 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:34.800269 29677 tablet_replica.cc:333] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04: stopping tablet replica
I20260317 21:03:34.800354 29677 raft_consensus.cc:2243] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:03:34.800439 29677 raft_consensus.cc:2272] T ca99df8523cd423b9247c879f4eedd90 P 15476ec7a2f24a4abbc60a26d9e30e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:34.802953 29677 tablet_server.cc:195] TabletServer@127.28.251.67:0 shutdown complete.
I20260317 21:03:34.803857 29677 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20260317 21:03:34.803995 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:34.805502 30999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:34.805546 31000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:34.805549 31002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:34.805781 29677 server_base.cc:1060] running on GCE node
I20260317 21:03:34.805903 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:34.805944 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:34.805958 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781414805958 us; error 0 us; skew 500 ppm
I20260317 21:03:34.806504 29677 webserver.cc:492] Webserver started at http://127.28.251.65:37375/ using document root <none> and password file <none>
I20260317 21:03:34.806571 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:34.806598 29677 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260317 21:03:34.807355 29677 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:34.808810 31010 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:34.809062 29677 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:34.809146 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-0-root
uuid: "364faf0bbe6c4665b365c26abc0824c4"
format_stamp: "Formatted at 2026-03-17 21:03:34 on dist-test-slave-tm0n"
I20260317 21:03:34.809249 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1773781378956650-29677-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5431
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:34.823204 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:34.823482 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:34.823690 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:34.824671 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260317 21:03:34.824726 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:34.825304 29677 ts_tablet_manager.cc:616] Registered 1 tablets
I20260317 21:03:34.825356 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:34.825412 31017 tablet_bootstrap.cc:492] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: Bootstrap starting.
I20260317 21:03:34.827512 31017 tablet_bootstrap.cc:492] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 21:03:34.827777 31017 tablet_bootstrap.cc:492] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: Bootstrap complete.
I20260317 21:03:34.827895 31017 ts_tablet_manager.cc:1403] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260317 21:03:34.828049 31017 raft_consensus.cc:359] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.828133 31017 raft_consensus.cc:740] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364faf0bbe6c4665b365c26abc0824c4, State: Initialized, Role: FOLLOWER
I20260317 21:03:34.828194 31017 consensus_queue.cc:260] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 1.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "fee0cb2d84a6473c8038690c01b873c9" member_type: VOTER last_known_addr { host: "127.28.251.66" port: 42085 } } peers { permanent_uuid: "15476ec7a2f24a4abbc60a26d9e30e04" member_type: VOTER last_known_addr { host: "127.28.251.67" port: 45017 } } peers { permanent_uuid: "364faf0bbe6c4665b365c26abc0824c4" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 46151 } }
I20260317 21:03:34.828377 31017 ts_tablet_manager.cc:1434] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20260317 21:03:34.829128 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.65:46151
I20260317 21:03:34.831230 31076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.65:46151 every 8 connection(s)
W20260317 21:03:34.832047 31077 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260317 21:03:34.837625 31030 tablet_service.cc:3018] Rejecting scan request for tablet ca99df8523cd423b9247c879f4eedd90: Uninitialized: clean time has not yet been initialized
I20260317 21:03:34.837940 29677 timestamp_advancement-itest.cc:447] Got response: error { code: TABLET_NOT_RUNNING status { code: UNINITIALIZED message: "clean time has not yet been initialized" } }
I20260317 21:03:34.838261 29677 tablet_server.cc:178] TabletServer@127.28.251.65:46151 shutting down...
I20260317 21:03:35.081792 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:35.082051 29677 tablet_replica.cc:333] stopping tablet replica
I20260317 21:03:35.082141 29677 raft_consensus.cc:2243] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:03:35.082221 29677 raft_consensus.cc:2272] T ca99df8523cd423b9247c879f4eedd90 P 364faf0bbe6c4665b365c26abc0824c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:35.094275 29677 tablet_server.cc:195] TabletServer@127.28.251.65:46151 shutdown complete.
[ OK ] TimestampAdvancementITest.Kudu2463Test (910 ms)
[ RUN ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp
I20260317 21:03:35.099547 29677 timestamp_advancement-itest.cc:460] Running with 1 replicas
I20260317 21:03:35.099700 29677 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.251.126:45431
I20260317 21:03:35.099838 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:35.101392 31081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:35.101462 29677 server_base.cc:1060] running on GCE node
W20260317 21:03:35.101439 31084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:35.101406 31082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:35.101903 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:35.101943 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:35.101955 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781415101956 us; error 0 us; skew 500 ppm
I20260317 21:03:35.102555 29677 webserver.cc:492] Webserver started at http://127.28.251.126:45965/ using document root <none> and password file <none>
I20260317 21:03:35.102666 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:35.102747 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:35.102804 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:35.103091 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/master-0-root/instance:
uuid: "fa140e1e3ad542b888e438c9dcca4d90"
format_stamp: "Formatted at 2026-03-17 21:03:35 on dist-test-slave-tm0n"
I20260317 21:03:35.103981 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:35.104473 31089 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.104622 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.104673 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/master-0-root
uuid: "fa140e1e3ad542b888e438c9dcca4d90"
format_stamp: "Formatted at 2026-03-17 21:03:35 on dist-test-slave-tm0n"
I20260317 21:03:35.104705 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:35.115753 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:35.115981 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:35.118836 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.126:45431
I20260317 21:03:35.119560 31141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.126:45431 every 8 connection(s)
I20260317 21:03:35.119686 31142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:03:35.120771 31142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90: Bootstrap starting.
I20260317 21:03:35.121096 31142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90: Neither blocks nor log segments found. Creating new log.
I20260317 21:03:35.121680 31142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90: No bootstrap required, opened a new log
I20260317 21:03:35.121824 31142 raft_consensus.cc:359] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER }
I20260317 21:03:35.121884 31142 raft_consensus.cc:385] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:03:35.121899 31142 raft_consensus.cc:740] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa140e1e3ad542b888e438c9dcca4d90, State: Initialized, Role: FOLLOWER
I20260317 21:03:35.121958 31142 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER }
I20260317 21:03:35.121996 31142 raft_consensus.cc:399] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 21:03:35.122022 31142 raft_consensus.cc:493] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 21:03:35.122048 31142 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:03:35.122560 31142 raft_consensus.cc:515] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER }
I20260317 21:03:35.122612 31142 leader_election.cc:304] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa140e1e3ad542b888e438c9dcca4d90; no voters:
I20260317 21:03:35.122730 31142 leader_election.cc:290] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260317 21:03:35.122859 31145 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:03:35.122895 31142 sys_catalog.cc:565] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:03:35.123065 31145 raft_consensus.cc:697] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 1 LEADER]: Becoming Leader. State: Replica: fa140e1e3ad542b888e438c9dcca4d90, State: Running, Role: LEADER
I20260317 21:03:35.123160 31145 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER }
I20260317 21:03:35.123440 31146 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa140e1e3ad542b888e438c9dcca4d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER } }
I20260317 21:03:35.123441 31147 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa140e1e3ad542b888e438c9dcca4d90. Latest consensus state: current_term: 1 leader_uuid: "fa140e1e3ad542b888e438c9dcca4d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER } }
I20260317 21:03:35.123611 31147 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [sys.catalog]: This master's current role is: LEADER
I20260317 21:03:35.123795 31146 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [sys.catalog]: This master's current role is: LEADER
I20260317 21:03:35.123803 31155 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:03:35.123996 31155 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:03:35.124400 29677 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260317 21:03:35.124616 31155 catalog_manager.cc:1357] Generated new cluster ID: 305cc09fb2194e2da6ed3a26c55d2a96
I20260317 21:03:35.124665 31155 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:03:35.140496 31155 catalog_manager.cc:1380] Generated new certificate authority record
I20260317 21:03:35.140903 31155 catalog_manager.cc:1514] Loading token signing keys...
I20260317 21:03:35.147754 31155 catalog_manager.cc:6044] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90: Generated new TSK 0
I20260317 21:03:35.147847 31155 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:03:35.156399 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:35.158008 31163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:35.158046 31164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:35.158068 29677 server_base.cc:1060] running on GCE node
W20260317 21:03:35.158042 31166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:35.158393 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:35.158447 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:35.158470 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781415158469 us; error 0 us; skew 500 ppm
I20260317 21:03:35.159055 29677 webserver.cc:492] Webserver started at http://127.28.251.65:44641/ using document root <none> and password file <none>
I20260317 21:03:35.159162 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:35.159224 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:35.159294 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:35.159616 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-0-root/instance:
uuid: "cf2dd2e6719b4f23ae6254649840004b"
format_stamp: "Formatted at 2026-03-17 21:03:35 on dist-test-slave-tm0n"
I20260317 21:03:35.160676 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:35.161140 31171 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.161319 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.161382 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-0-root
uuid: "cf2dd2e6719b4f23ae6254649840004b"
format_stamp: "Formatted at 2026-03-17 21:03:35 on dist-test-slave-tm0n"
I20260317 21:03:35.161433 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:35.184866 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:35.185132 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:35.185359 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:35.185592 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:03:35.185638 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.185678 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:03:35.185708 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.188678 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.65:42919
I20260317 21:03:35.188711 31234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.65:42919 every 8 connection(s)
I20260317 21:03:35.192474 31235 heartbeater.cc:344] Connected to a master server at 127.28.251.126:45431
I20260317 21:03:35.192533 31235 heartbeater.cc:461] Registering TS with master...
I20260317 21:03:35.192610 31235 heartbeater.cc:507] Master 127.28.251.126:45431 requested a full tablet report, sending...
I20260317 21:03:35.192831 31106 ts_manager.cc:194] Registered new tserver with Master: cf2dd2e6719b4f23ae6254649840004b (127.28.251.65:42919)
I20260317 21:03:35.193320 29677 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004432644s
I20260317 21:03:35.193579 31106 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39394
I20260317 21:03:35.205646 31106 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39428:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:03:35.210155 31199 tablet_service.cc:1511] Processing CreateTablet for tablet 354cb3910347457589ac079f0722b9a7 (DEFAULT_TABLE table=test-workload [id=4f81c55711a242619b29c8eba45c5f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:03:35.210291 31199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 354cb3910347457589ac079f0722b9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:03:35.211643 31253 tablet_bootstrap.cc:492] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: Bootstrap starting.
I20260317 21:03:35.212013 31253 tablet_bootstrap.cc:654] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: Neither blocks nor log segments found. Creating new log.
I20260317 21:03:35.212572 31253 tablet_bootstrap.cc:492] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: No bootstrap required, opened a new log
I20260317 21:03:35.212637 31253 ts_tablet_manager.cc:1403] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:35.212764 31253 raft_consensus.cc:359] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2dd2e6719b4f23ae6254649840004b" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 42919 } }
I20260317 21:03:35.212908 31253 raft_consensus.cc:385] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:03:35.212968 31253 raft_consensus.cc:740] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2dd2e6719b4f23ae6254649840004b, State: Initialized, Role: FOLLOWER
I20260317 21:03:35.213034 31253 consensus_queue.cc:260] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2dd2e6719b4f23ae6254649840004b" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 42919 } }
I20260317 21:03:35.213099 31253 raft_consensus.cc:399] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 21:03:35.213127 31253 raft_consensus.cc:493] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 21:03:35.213150 31253 raft_consensus.cc:3060] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:03:35.213744 31253 raft_consensus.cc:515] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2dd2e6719b4f23ae6254649840004b" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 42919 } }
I20260317 21:03:35.213805 31253 leader_election.cc:304] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf2dd2e6719b4f23ae6254649840004b; no voters:
I20260317 21:03:35.213938 31253 leader_election.cc:290] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260317 21:03:35.214031 31255 raft_consensus.cc:2804] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:03:35.214100 31253 ts_tablet_manager.cc:1434] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260317 21:03:35.214154 31235 heartbeater.cc:499] Master 127.28.251.126:45431 was elected leader, sending a full tablet report...
I20260317 21:03:35.214217 31255 raft_consensus.cc:697] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 1 LEADER]: Becoming Leader. State: Replica: cf2dd2e6719b4f23ae6254649840004b, State: Running, Role: LEADER
I20260317 21:03:35.214260 31255 consensus_queue.cc:237] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2dd2e6719b4f23ae6254649840004b" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 42919 } }
I20260317 21:03:35.214720 31106 catalog_manager.cc:5671] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b reported cstate change: term changed from 0 to 1, leader changed from <none> to cf2dd2e6719b4f23ae6254649840004b (127.28.251.65). New cstate: current_term: 1 leader_uuid: "cf2dd2e6719b4f23ae6254649840004b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2dd2e6719b4f23ae6254649840004b" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 42919 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:35.216121 29677 tablet_server.cc:178] TabletServer@127.28.251.65:0 shutting down...
I20260317 21:03:35.218667 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:35.218838 29677 tablet_replica.cc:333] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: stopping tablet replica
I20260317 21:03:35.218904 29677 raft_consensus.cc:2243] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:03:35.218953 29677 raft_consensus.cc:2272] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:35.230947 29677 tablet_server.cc:195] TabletServer@127.28.251.65:0 shutdown complete.
I20260317 21:03:35.232023 29677 master.cc:561] Master@127.28.251.126:45431 shutting down...
I20260317 21:03:35.234886 29677 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:03:35.234956 29677 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:35.235028 29677 tablet_replica.cc:333] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90: stopping tablet replica
I20260317 21:03:35.247301 29677 master.cc:583] Master@127.28.251.126:45431 shutdown complete.
I20260317 21:03:35.249217 29677 timestamp_advancement-itest.cc:460] Running with 3 replicas
I20260317 21:03:35.249428 29677 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.251.126:44105
I20260317 21:03:35.249586 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:35.250756 31260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:35.250867 31263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:35.250967 29677 server_base.cc:1060] running on GCE node
W20260317 21:03:35.251037 31261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:35.251186 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:35.251225 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:35.251246 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781415251247 us; error 0 us; skew 500 ppm
I20260317 21:03:35.251847 29677 webserver.cc:492] Webserver started at http://127.28.251.126:40263/ using document root <none> and password file <none>
I20260317 21:03:35.251953 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:35.251994 29677 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260317 21:03:35.252557 29677 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260317 21:03:35.252988 31268 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.253146 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.253315 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/master-0-root
uuid: "fa140e1e3ad542b888e438c9dcca4d90"
format_stamp: "Formatted at 2026-03-17 21:03:35 on dist-test-slave-tm0n"
I20260317 21:03:35.253394 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:35.261394 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:35.261615 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:35.264424 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.126:44105
I20260317 21:03:35.265421 31320 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.126:44105 every 8 connection(s)
I20260317 21:03:35.266000 31321 sys_catalog.cc:263] Verifying existing consensus state
I20260317 21:03:35.266391 31321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90: Bootstrap starting.
I20260317 21:03:35.267869 31321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 21:03:35.268122 31321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90: Bootstrap complete.
I20260317 21:03:35.268335 31321 raft_consensus.cc:359] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER }
I20260317 21:03:35.268404 31321 raft_consensus.cc:740] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa140e1e3ad542b888e438c9dcca4d90, State: Initialized, Role: FOLLOWER
I20260317 21:03:35.268463 31321 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER }
I20260317 21:03:35.268497 31321 raft_consensus.cc:399] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 21:03:35.268532 31321 raft_consensus.cc:493] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 21:03:35.268546 31321 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:03:35.269038 31321 raft_consensus.cc:515] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER }
I20260317 21:03:35.269088 31321 leader_election.cc:304] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa140e1e3ad542b888e438c9dcca4d90; no voters:
I20260317 21:03:35.269233 31321 leader_election.cc:290] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260317 21:03:35.269296 31324 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 21:03:35.269409 31321 sys_catalog.cc:565] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:03:35.269413 31324 raft_consensus.cc:697] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 2 LEADER]: Becoming Leader. State: Replica: fa140e1e3ad542b888e438c9dcca4d90, State: Running, Role: LEADER
I20260317 21:03:35.269598 31324 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER }
I20260317 21:03:35.269881 31326 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa140e1e3ad542b888e438c9dcca4d90. Latest consensus state: current_term: 2 leader_uuid: "fa140e1e3ad542b888e438c9dcca4d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER } }
I20260317 21:03:35.269857 31325 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fa140e1e3ad542b888e438c9dcca4d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa140e1e3ad542b888e438c9dcca4d90" member_type: VOTER } }
I20260317 21:03:35.269965 31326 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [sys.catalog]: This master's current role is: LEADER
I20260317 21:03:35.269984 31325 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [sys.catalog]: This master's current role is: LEADER
I20260317 21:03:35.270164 31331 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:03:35.270402 31331 catalog_manager.cc:679] Loaded metadata for table test-workload [id=4f81c55711a242619b29c8eba45c5f07]
I20260317 21:03:35.270530 31331 tablet_loader.cc:96] loaded metadata for tablet 354cb3910347457589ac079f0722b9a7 (table test-workload [id=4f81c55711a242619b29c8eba45c5f07])
I20260317 21:03:35.270576 31331 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:03:35.270654 31331 catalog_manager.cc:1269] Loaded cluster ID: 305cc09fb2194e2da6ed3a26c55d2a96
I20260317 21:03:35.270690 31331 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:03:35.270936 29677 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260317 21:03:35.271497 31331 catalog_manager.cc:1514] Loading token signing keys...
I20260317 21:03:35.271591 31331 catalog_manager.cc:6055] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90: Loaded TSK: 0
I20260317 21:03:35.271663 31331 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:03:35.272229 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:35.273658 31344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:35.273685 31341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:35.273773 31342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:35.274031 29677 server_base.cc:1060] running on GCE node
I20260317 21:03:35.274127 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:35.274163 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:35.274180 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781415274181 us; error 0 us; skew 500 ppm
I20260317 21:03:35.274850 29677 webserver.cc:492] Webserver started at http://127.28.251.65:39491/ using document root <none> and password file <none>
I20260317 21:03:35.274933 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:35.274991 29677 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260317 21:03:35.275631 29677 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.276242 31349 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.276439 29677 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:35.276487 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-0-root
uuid: "cf2dd2e6719b4f23ae6254649840004b"
format_stamp: "Formatted at 2026-03-17 21:03:35 on dist-test-slave-tm0n"
I20260317 21:03:35.276561 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:35.291244 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:35.291538 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:35.291702 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:35.292063 31356 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260317 21:03:35.292742 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260317 21:03:35.292809 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260317 21:03:35.292858 29677 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260317 21:03:35.293360 29677 ts_tablet_manager.cc:616] Registered 1 tablets
I20260317 21:03:35.293411 31356 tablet_bootstrap.cc:492] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: Bootstrap starting.
I20260317 21:03:35.293423 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:35.294870 31356 tablet_bootstrap.cc:492] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 21:03:35.295164 31356 tablet_bootstrap.cc:492] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: Bootstrap complete.
I20260317 21:03:35.295289 31356 ts_tablet_manager.cc:1403] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260317 21:03:35.295440 31356 raft_consensus.cc:359] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2dd2e6719b4f23ae6254649840004b" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 42919 } }
I20260317 21:03:35.295529 31356 raft_consensus.cc:740] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2dd2e6719b4f23ae6254649840004b, State: Initialized, Role: FOLLOWER
I20260317 21:03:35.295588 31356 consensus_queue.cc:260] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2dd2e6719b4f23ae6254649840004b" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 42919 } }
I20260317 21:03:35.295648 31356 raft_consensus.cc:399] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260317 21:03:35.295676 31356 raft_consensus.cc:493] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260317 21:03:35.295706 31356 raft_consensus.cc:3060] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:03:35.296663 31356 raft_consensus.cc:515] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2dd2e6719b4f23ae6254649840004b" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 42919 } }
I20260317 21:03:35.296761 31356 leader_election.cc:304] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf2dd2e6719b4f23ae6254649840004b; no voters:
I20260317 21:03:35.296872 31356 leader_election.cc:290] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260317 21:03:35.296933 31410 raft_consensus.cc:2804] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 2 FOLLOWER]: Leader election won for term 2
I20260317 21:03:35.297008 31356 ts_tablet_manager.cc:1434] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260317 21:03:35.297119 31410 raft_consensus.cc:697] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 2 LEADER]: Becoming Leader. State: Replica: cf2dd2e6719b4f23ae6254649840004b, State: Running, Role: LEADER
I20260317 21:03:35.297202 31410 consensus_queue.cc:237] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2dd2e6719b4f23ae6254649840004b" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 42919 } }
I20260317 21:03:35.297292 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.65:39157
I20260317 21:03:35.297469 31416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.65:39157 every 8 connection(s)
I20260317 21:03:35.297924 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:35.299515 31423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:35.299566 31424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:35.299779 31426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:35.300088 29677 server_base.cc:1060] running on GCE node
I20260317 21:03:35.300256 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:35.300303 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:35.300324 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781415300324 us; error 0 us; skew 500 ppm
I20260317 21:03:35.301407 29677 webserver.cc:492] Webserver started at http://127.28.251.66:33339/ using document root <none> and password file <none>
I20260317 21:03:35.301506 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:35.301564 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:35.301617 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:35.301949 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-1-root/instance:
uuid: "ea51bc7ac0c648dea42c7233c613af90"
format_stamp: "Formatted at 2026-03-17 21:03:35 on dist-test-slave-tm0n"
I20260317 21:03:35.302002 31418 heartbeater.cc:344] Connected to a master server at 127.28.251.126:44105
I20260317 21:03:35.302052 31418 heartbeater.cc:461] Registering TS with master...
I20260317 21:03:35.302141 31418 heartbeater.cc:507] Master 127.28.251.126:44105 requested a full tablet report, sending...
I20260317 21:03:35.302466 31285 ts_manager.cc:194] Registered new tserver with Master: cf2dd2e6719b4f23ae6254649840004b (127.28.251.65:39157)
I20260317 21:03:35.302510 31285 catalog_manager.cc:5671] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "cf2dd2e6719b4f23ae6254649840004b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2dd2e6719b4f23ae6254649840004b" member_type: VOTER last_known_addr { host: "127.28.251.65" port: 42919 } health_report { overall_health: HEALTHY } } }
I20260317 21:03:35.303098 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260317 21:03:35.303785 31432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.303889 31285 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39512
I20260317 21:03:35.303990 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260317 21:03:35.304065 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-1-root
uuid: "ea51bc7ac0c648dea42c7233c613af90"
format_stamp: "Formatted at 2026-03-17 21:03:35 on dist-test-slave-tm0n"
I20260317 21:03:35.304149 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:35.304385 31418 heartbeater.cc:499] Master 127.28.251.126:44105 was elected leader, sending a full tablet report...
I20260317 21:03:35.312824 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:35.313120 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:35.313376 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:35.313612 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:03:35.313665 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.313707 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:03:35.313735 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.316933 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.66:35733
I20260317 21:03:35.317009 31495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.66:35733 every 8 connection(s)
I20260317 21:03:35.317451 29677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:03:35.318967 31499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:03:35.319013 31500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:35.319183 29677 server_base.cc:1060] running on GCE node
W20260317 21:03:35.319023 31502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:03:35.319427 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260317 21:03:35.319481 29677 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260317 21:03:35.319504 29677 hybrid_clock.cc:648] HybridClock initialized: now 1773781415319504 us; error 0 us; skew 500 ppm
I20260317 21:03:35.320187 29677 webserver.cc:492] Webserver started at http://127.28.251.67:42301/ using document root <none> and password file <none>
I20260317 21:03:35.320288 29677 fs_manager.cc:362] Metadata directory not provided
I20260317 21:03:35.320348 29677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:03:35.320432 29677 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:03:35.320786 29677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-2-root/instance:
uuid: "105f3f74990d4891a1e999d560092c09"
format_stamp: "Formatted at 2026-03-17 21:03:35 on dist-test-slave-tm0n"
I20260317 21:03:35.320966 31496 heartbeater.cc:344] Connected to a master server at 127.28.251.126:44105
I20260317 21:03:35.321030 31496 heartbeater.cc:461] Registering TS with master...
I20260317 21:03:35.321195 31496 heartbeater.cc:507] Master 127.28.251.126:44105 requested a full tablet report, sending...
I20260317 21:03:35.321483 31285 ts_manager.cc:194] Registered new tserver with Master: ea51bc7ac0c648dea42c7233c613af90 (127.28.251.66:35733)
I20260317 21:03:35.321945 29677 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260317 21:03:35.321949 31285 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39524
I20260317 21:03:35.322520 31507 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.322674 29677 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260317 21:03:35.322718 29677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-2-root
uuid: "105f3f74990d4891a1e999d560092c09"
format_stamp: "Formatted at 2026-03-17 21:03:35 on dist-test-slave-tm0n"
I20260317 21:03:35.322777 29677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasklDlrCj/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1773781378956650-29677-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:03:35.328024 29677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:03:35.328320 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:03:35.328491 29677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:03:35.328696 29677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:03:35.328725 29677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.328747 29677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:03:35.328760 29677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:03:35.332306 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.67:41609
I20260317 21:03:35.332353 31570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.251.67:41609 every 8 connection(s)
I20260317 21:03:35.335790 31571 heartbeater.cc:344] Connected to a master server at 127.28.251.126:44105
I20260317 21:03:35.335858 31571 heartbeater.cc:461] Registering TS with master...
I20260317 21:03:35.335994 31571 heartbeater.cc:507] Master 127.28.251.126:44105 requested a full tablet report, sending...
I20260317 21:03:35.336264 31285 ts_manager.cc:194] Registered new tserver with Master: 105f3f74990d4891a1e999d560092c09 (127.28.251.67:41609)
I20260317 21:03:35.336634 31285 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39540
I20260317 21:03:35.336971 29677 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004453109s
I20260317 21:03:35.349521 29677 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20260317 21:03:35.349639 29677 tablet_server.cc:178] TabletServer@127.28.251.65:0 shutting down...
I20260317 21:03:35.352242 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:35.352524 29677 tablet_replica.cc:333] stopping tablet replica
I20260317 21:03:35.352614 29677 raft_consensus.cc:2243] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 2 LEADER]: Raft consensus shutting down.
I20260317 21:03:35.352685 29677 raft_consensus.cc:2272] T 354cb3910347457589ac079f0722b9a7 P cf2dd2e6719b4f23ae6254649840004b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:35.364771 29677 tablet_server.cc:195] TabletServer@127.28.251.65:0 shutdown complete.
I20260317 21:03:35.365754 29677 tablet_server.cc:178] TabletServer@127.28.251.66:0 shutting down...
I20260317 21:03:35.368544 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:35.380070 29677 tablet_server.cc:195] TabletServer@127.28.251.66:0 shutdown complete.
I20260317 21:03:35.381105 29677 tablet_server.cc:178] TabletServer@127.28.251.67:0 shutting down...
I20260317 21:03:35.384172 29677 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260317 21:03:35.395979 29677 tablet_server.cc:195] TabletServer@127.28.251.67:0 shutdown complete.
I20260317 21:03:35.396896 29677 master.cc:561] Master@127.28.251.126:44105 shutting down...
I20260317 21:03:35.399735 29677 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 2 LEADER]: Raft consensus shutting down.
I20260317 21:03:35.399824 29677 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fa140e1e3ad542b888e438c9dcca4d90 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 21:03:35.399894 29677 tablet_replica.cc:333] stopping tablet replica
I20260317 21:03:35.412142 29677 master.cc:583] Master@127.28.251.126:44105 shutdown complete.
[ OK ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp (317 ms)
[----------] 4 tests from TimestampAdvancementITest (36455 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (36455 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
1 FAILED TEST
I20260317 21:03:35.416783 29677 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5349: suppressed but not reported on 7 messages since previous log ~1 seconds ago
I20260317 21:03:35.416899 29677 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 6 messages since previous log ~1 seconds ago
I20260317 21:03:35.416931 29677 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20260317 21:03:35.417001 29677 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/maintenance_manager.cc:346: suppressed but not reported on 559 messages since previous log ~36 seconds ago