[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TimestampAdvancementITest
[ RUN ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260501 08:10:50.601069 1100 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.19.62:45747
I20260501 08:10:50.601446 1100 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 08:10:50.601620 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:10:50.605580 1106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:10:50.605584 1108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:10:50.605665 1105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:10:50.605741 1100 server_base.cc:1061] running on GCE node
I20260501 08:10:50.605933 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:10:50.605971 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:10:50.605993 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623050605994 us; error 0 us; skew 500 ppm
I20260501 08:10:50.607110 1100 webserver.cc:492] Webserver started at http://127.1.19.62:36879/ using document root <none> and password file <none>
I20260501 08:10:50.607290 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:10:50.607336 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:10:50.607426 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:10:50.608501 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/master-0-root/instance:
uuid: "42b281f3ef9c42f6a07d3ec8932cf8f8"
format_stamp: "Formatted at 2026-05-01 08:10:50 on dist-test-slave-pv7w"
I20260501 08:10:50.609772 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 08:10:50.610431 1113 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.610644 1100 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260501 08:10:50.610709 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/master-0-root
uuid: "42b281f3ef9c42f6a07d3ec8932cf8f8"
format_stamp: "Formatted at 2026-05-01 08:10:50 on dist-test-slave-pv7w"
I20260501 08:10:50.610760 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:10:50.624018 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:10:50.624404 1100 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 08:10:50.624512 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:10:50.624886 1119 maintenance_manager.cc:346] Maintenance manager is disabled (check --enable_maintenance_manager).
I20260501 08:10:50.628993 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.62:45747
I20260501 08:10:50.629083 1165 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.62:45747 every 8 connection(s)
I20260501 08:10:50.629652 1166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:10:50.631978 1166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8: Bootstrap starting.
I20260501 08:10:50.632828 1166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8: Neither blocks nor log segments found. Creating new log.
I20260501 08:10:50.633081 1166 log.cc:826] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8: Log is configured to *not* fsync() on all Append() calls
I20260501 08:10:50.633625 1166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8: No bootstrap required, opened a new log
I20260501 08:10:50.634786 1166 raft_consensus.cc:359] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b281f3ef9c42f6a07d3ec8932cf8f8" member_type: VOTER }
I20260501 08:10:50.634876 1166 raft_consensus.cc:385] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:10:50.634894 1166 raft_consensus.cc:740] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b281f3ef9c42f6a07d3ec8932cf8f8, State: Initialized, Role: FOLLOWER
I20260501 08:10:50.635088 1166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b281f3ef9c42f6a07d3ec8932cf8f8" member_type: VOTER }
I20260501 08:10:50.635154 1166 raft_consensus.cc:399] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 08:10:50.635174 1166 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 08:10:50.635241 1166 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:10:50.635742 1166 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b281f3ef9c42f6a07d3ec8932cf8f8" member_type: VOTER }
I20260501 08:10:50.635820 1166 leader_election.cc:304] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42b281f3ef9c42f6a07d3ec8932cf8f8; no voters:
I20260501 08:10:50.635975 1166 leader_election.cc:290] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 08:10:50.636039 1169 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 08:10:50.636227 1166 sys_catalog.cc:565] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 08:10:50.636272 1169 raft_consensus.cc:697] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 1 LEADER]: Becoming Leader. State: Replica: 42b281f3ef9c42f6a07d3ec8932cf8f8, State: Running, Role: LEADER
I20260501 08:10:50.636416 1169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b281f3ef9c42f6a07d3ec8932cf8f8" member_type: VOTER }
I20260501 08:10:50.636750 1170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42b281f3ef9c42f6a07d3ec8932cf8f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b281f3ef9c42f6a07d3ec8932cf8f8" member_type: VOTER } }
I20260501 08:10:50.636839 1170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [sys.catalog]: This master's current role is: LEADER
I20260501 08:10:50.636746 1171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42b281f3ef9c42f6a07d3ec8932cf8f8. Latest consensus state: current_term: 1 leader_uuid: "42b281f3ef9c42f6a07d3ec8932cf8f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b281f3ef9c42f6a07d3ec8932cf8f8" member_type: VOTER } }
I20260501 08:10:50.637005 1171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [sys.catalog]: This master's current role is: LEADER
I20260501 08:10:50.637271 1176 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 08:10:50.637660 1176 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 08:10:50.637804 1100 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260501 08:10:50.639215 1176 catalog_manager.cc:1357] Generated new cluster ID: 3626e1bf89704b25958193348b7a1a34
I20260501 08:10:50.639286 1176 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 08:10:50.647928 1176 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 08:10:50.648523 1176 catalog_manager.cc:1514] Loading token signing keys...
I20260501 08:10:50.657632 1176 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8: Generated new TSK 0
I20260501 08:10:50.657845 1176 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 08:10:50.669984 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260501 08:10:50.670205 1100 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260501 08:10:50.671798 1188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:10:50.671841 1189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:10:50.672060 1100 server_base.cc:1061] running on GCE node
W20260501 08:10:50.671856 1191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:10:50.672329 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:10:50.672362 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:10:50.672375 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623050672375 us; error 0 us; skew 500 ppm
I20260501 08:10:50.673038 1100 webserver.cc:492] Webserver started at http://127.1.19.1:38401/ using document root <none> and password file <none>
I20260501 08:10:50.673148 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:10:50.673213 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:10:50.673280 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:10:50.673566 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-0-root/instance:
uuid: "da6eb0fe0b9f41dc9adc9114840069da"
format_stamp: "Formatted at 2026-05-01 08:10:50 on dist-test-slave-pv7w"
I20260501 08:10:50.674561 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260501 08:10:50.675072 1196 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.675195 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260501 08:10:50.675246 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-0-root
uuid: "da6eb0fe0b9f41dc9adc9114840069da"
format_stamp: "Formatted at 2026-05-01 08:10:50 on dist-test-slave-pv7w"
I20260501 08:10:50.675302 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:10:50.704505 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:10:50.705142 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:10:50.705402 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:10:50.705752 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:10:50.705790 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.705816 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:10:50.705854 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.710729 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.1:36841
I20260501 08:10:50.710803 1259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.1:36841 every 8 connection(s)
I20260501 08:10:50.711273 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260501 08:10:50.711440 1100 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260501 08:10:50.712839 1265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:10:50.712896 1267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:10:50.713167 1100 server_base.cc:1061] running on GCE node
W20260501 08:10:50.713889 1264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:10:50.714139 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:10:50.714198 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:10:50.714231 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623050714231 us; error 0 us; skew 500 ppm
I20260501 08:10:50.714931 1100 webserver.cc:492] Webserver started at http://127.1.19.2:35065/ using document root <none> and password file <none>
I20260501 08:10:50.715040 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:10:50.715106 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:10:50.715142 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:10:50.715519 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-1-root/instance:
uuid: "4b9150123ff048d08f1d65634bf2425a"
format_stamp: "Formatted at 2026-05-01 08:10:50 on dist-test-slave-pv7w"
I20260501 08:10:50.716204 1260 heartbeater.cc:344] Connected to a master server at 127.1.19.62:45747
I20260501 08:10:50.716284 1260 heartbeater.cc:461] Registering TS with master...
I20260501 08:10:50.716436 1260 heartbeater.cc:507] Master 127.1.19.62:45747 requested a full tablet report, sending...
I20260501 08:10:50.716769 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260501 08:10:50.716851 1130 ts_manager.cc:194] Registered new tserver with Master: da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:50.717368 1272 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.717548 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260501 08:10:50.717607 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-1-root
uuid: "4b9150123ff048d08f1d65634bf2425a"
format_stamp: "Formatted at 2026-05-01 08:10:50 on dist-test-slave-pv7w"
I20260501 08:10:50.717664 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:10:50.717716 1130 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42990
I20260501 08:10:50.749121 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:10:50.749447 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:10:50.749670 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:10:50.750001 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:10:50.750056 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.750082 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:10:50.750139 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.754703 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.2:43689
I20260501 08:10:50.754812 1335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.2:43689 every 8 connection(s)
I20260501 08:10:50.755173 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260501 08:10:50.755301 1100 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260501 08:10:50.756682 1342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:10:50.756830 1340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:10:50.757023 1100 server_base.cc:1061] running on GCE node
W20260501 08:10:50.758356 1339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:10:50.758569 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:10:50.758608 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:10:50.758626 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623050758626 us; error 0 us; skew 500 ppm
I20260501 08:10:50.759225 1100 webserver.cc:492] Webserver started at http://127.1.19.3:42373/ using document root <none> and password file <none>
I20260501 08:10:50.759248 1336 heartbeater.cc:344] Connected to a master server at 127.1.19.62:45747
I20260501 08:10:50.759305 1336 heartbeater.cc:461] Registering TS with master...
I20260501 08:10:50.759313 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:10:50.759459 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:10:50.759516 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:10:50.759544 1336 heartbeater.cc:507] Master 127.1.19.62:45747 requested a full tablet report, sending...
I20260501 08:10:50.759801 1130 ts_manager.cc:194] Registered new tserver with Master: 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689)
I20260501 08:10:50.759876 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-2-root/instance:
uuid: "2be24db2181749b9be4f7f93ba03e486"
format_stamp: "Formatted at 2026-05-01 08:10:50 on dist-test-slave-pv7w"
I20260501 08:10:50.760337 1130 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42994
I20260501 08:10:50.761034 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260501 08:10:50.761823 1347 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.762020 1100 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260501 08:10:50.762084 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-2-root
uuid: "2be24db2181749b9be4f7f93ba03e486"
format_stamp: "Formatted at 2026-05-01 08:10:50 on dist-test-slave-pv7w"
I20260501 08:10:50.762157 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:10:50.772864 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:10:50.773149 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:10:50.773339 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:10:50.773561 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:10:50.773607 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.773649 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:10:50.773676 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.777575 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.3:46049
I20260501 08:10:50.777642 1410 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.3:46049 every 8 connection(s)
I20260501 08:10:50.784044 1411 heartbeater.cc:344] Connected to a master server at 127.1.19.62:45747
I20260501 08:10:50.784142 1411 heartbeater.cc:461] Registering TS with master...
I20260501 08:10:50.784250 1411 heartbeater.cc:507] Master 127.1.19.62:45747 requested a full tablet report, sending...
I20260501 08:10:50.784541 1130 ts_manager.cc:194] Registered new tserver with Master: 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:50.784613 1100 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006791196s
I20260501 08:10:50.785108 1130 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43008
I20260501 08:10:50.801931 1130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43034:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 08:10:50.802279 1130 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 08:10:50.808359 1375 tablet_service.cc:1511] Processing CreateTablet for tablet 4d0191c11e464652ac0b4901a8d0fd06 (DEFAULT_TABLE table=test-workload [id=2d96fe1c540b47cb85720ed9934308f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 08:10:50.808365 1300 tablet_service.cc:1511] Processing CreateTablet for tablet 4d0191c11e464652ac0b4901a8d0fd06 (DEFAULT_TABLE table=test-workload [id=2d96fe1c540b47cb85720ed9934308f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 08:10:50.808622 1224 tablet_service.cc:1511] Processing CreateTablet for tablet 4d0191c11e464652ac0b4901a8d0fd06 (DEFAULT_TABLE table=test-workload [id=2d96fe1c540b47cb85720ed9934308f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 08:10:50.808641 1375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d0191c11e464652ac0b4901a8d0fd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:10:50.808646 1300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d0191c11e464652ac0b4901a8d0fd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:10:50.809968 1433 tablet_bootstrap.cc:492] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Bootstrap starting.
I20260501 08:10:50.808712 1224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d0191c11e464652ac0b4901a8d0fd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:10:50.810462 1433 tablet_bootstrap.cc:654] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Neither blocks nor log segments found. Creating new log.
I20260501 08:10:50.810791 1432 tablet_bootstrap.cc:492] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Bootstrap starting.
I20260501 08:10:50.811002 1433 tablet_bootstrap.cc:492] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: No bootstrap required, opened a new log
I20260501 08:10:50.811058 1433 ts_tablet_manager.cc:1403] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260501 08:10:50.811125 1432 tablet_bootstrap.cc:654] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Neither blocks nor log segments found. Creating new log.
I20260501 08:10:50.811178 1433 raft_consensus.cc:359] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:50.811264 1433 raft_consensus.cc:385] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:10:50.811301 1433 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Initialized, Role: FOLLOWER
I20260501 08:10:50.811355 1433 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:50.811448 1435 tablet_bootstrap.cc:492] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Bootstrap starting.
I20260501 08:10:50.811560 1433 ts_tablet_manager.cc:1434] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260501 08:10:50.811669 1432 tablet_bootstrap.cc:492] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: No bootstrap required, opened a new log
I20260501 08:10:50.811717 1432 ts_tablet_manager.cc:1403] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260501 08:10:50.811723 1336 heartbeater.cc:499] Master 127.1.19.62:45747 was elected leader, sending a full tablet report...
I20260501 08:10:50.811826 1432 raft_consensus.cc:359] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:50.811941 1432 raft_consensus.cc:385] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:10:50.811965 1432 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Initialized, Role: FOLLOWER
I20260501 08:10:50.812013 1432 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:50.812198 1432 ts_tablet_manager.cc:1434] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260501 08:10:50.812288 1411 heartbeater.cc:499] Master 127.1.19.62:45747 was elected leader, sending a full tablet report...
I20260501 08:10:50.812467 1435 tablet_bootstrap.cc:654] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Neither blocks nor log segments found. Creating new log.
I20260501 08:10:50.813337 1435 tablet_bootstrap.cc:492] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: No bootstrap required, opened a new log
I20260501 08:10:50.813395 1435 ts_tablet_manager.cc:1403] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260501 08:10:50.813545 1435 raft_consensus.cc:359] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:50.813629 1435 raft_consensus.cc:385] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:10:50.813669 1435 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Initialized, Role: FOLLOWER
I20260501 08:10:50.813722 1435 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:50.813970 1260 heartbeater.cc:499] Master 127.1.19.62:45747 was elected leader, sending a full tablet report...
I20260501 08:10:50.813957 1435 ts_tablet_manager.cc:1434] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 08:10:50.823576 1438 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 08:10:50.823683 1438 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:50.824009 1438 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:50.827245 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a" is_pre_election: true
I20260501 08:10:50.827245 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da" is_pre_election: true
I20260501 08:10:50.827428 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 0.
I20260501 08:10:50.827428 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 0.
I20260501 08:10:50.827657 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters:
I20260501 08:10:50.827769 1438 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 08:10:50.827806 1438 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:50.827821 1438 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:10:50.828500 1438 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:50.828649 1438 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 1 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:50.828760 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:50.828845 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:10:50.828858 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:50.829056 1234 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:10:50.829484 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 1.
I20260501 08:10:50.829707 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters:
I20260501 08:10:50.829854 1438 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 08:10:50.829896 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 1.
I20260501 08:10:50.830096 1438 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 1 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:50.830159 1438 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:50.830842 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 1 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:50.838092 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 1 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 08:10:50.838092 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 1 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 08:10:50.838433 1438 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 08:10:50.838573 1442 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 08:10:50.858076 1100 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20260501 08:10:51.060096 1437 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:51.060225 1437 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 1 FOLLOWER]: Advancing to term 2
I20260501 08:10:51.060969 1437 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.061199 1437 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 2 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:51.064849 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:51.064874 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:51.065162 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:51.065292 1437 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260501 08:10:51.083093 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 2 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 1. Current term is 2. Ops: []
I20260501 08:10:51.083513 1477 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 1.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20260501 08:10:51.083726 1438 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260501 08:10:51.083775 1438 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260501 08:10:51.083909 1438 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 1 LEADER]: Stepping down as leader of term 1
I20260501 08:10:51.083966 1438 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:51.084017 1438 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.084183 1438 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 1 FOLLOWER]: Advancing to term 2
I20260501 08:10:51.151590 1440 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:51.151700 1440 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 1 FOLLOWER]: Advancing to term 2
I20260501 08:10:51.152500 1440 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.152725 1440 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 2 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:51.155611 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:51.155706 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 2: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:51.155933 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:51.156641 1385 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 2.
I20260501 08:10:51.156828 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:51.156932 1440 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 2 FOLLOWER]: Leader election won for term 2
I20260501 08:10:51.157099 1440 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 2 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:51.157171 1440 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.157668 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 1 to 2, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 2 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:51.188742 1437 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:51.188860 1437 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 2 FOLLOWER]: Advancing to term 3
I20260501 08:10:51.189601 1437 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.189848 1437 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 3 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:51.190160 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:51.190155 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:51.190276 1385 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 2 FOLLOWER]: Advancing to term 3
I20260501 08:10:51.190866 1385 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 3.
I20260501 08:10:51.191071 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:51.191206 1437 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 3 FOLLOWER]: Leader election won for term 3
I20260501 08:10:51.191316 1437 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 3 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:51.191359 1437 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.215750 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 2 to 3, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 3 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:51.263734 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 3 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 2. Current term is 3. Ops: []
I20260501 08:10:51.264158 1440 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260501 08:10:51.264305 1440 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 3.41 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 40
I20260501 08:10:51.273844 1234 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 2 LEADER]: Stepping down as leader of term 2
I20260501 08:10:51.273959 1234 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:51.274025 1234 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.41, Last appended by leader: 41, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.274171 1234 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 2 FOLLOWER]: Advancing to term 3
I20260501 08:10:51.274189 1385 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 3 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20260501 08:10:51.274412 1437 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260501 08:10:51.275101 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 3 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 41. Preceding OpId from leader: term: 3 index: 41. (term mismatch)
I20260501 08:10:51.275175 1234 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 40
I20260501 08:10:51.275218 1234 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 2.41
I20260501 08:10:51.275250 1234 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:51.275492 1481 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260501 08:10:51.472159 1438 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:10:51.472270 1438 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 3 FOLLOWER]: Advancing to term 4
I20260501 08:10:51.478331 1438 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.478606 1438 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 4 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:51.478840 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:51.478930 1234 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 3 FOLLOWER]: Advancing to term 4
I20260501 08:10:51.478801 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:51.479075 1385 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 4 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 3. Current term is 4. Ops: []
I20260501 08:10:51.479306 1437 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 3.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20260501 08:10:51.479809 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 4.
I20260501 08:10:51.480017 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:51.480190 1438 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 4 FOLLOWER]: Leader election won for term 4
I20260501 08:10:51.480296 1438 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 4 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:51.480355 1438 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.480405 1234 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 4 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 3. Current term is 4. Ops: []
I20260501 08:10:51.480580 1481 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 3.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20260501 08:10:51.480800 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 3 to 4, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 4 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:51.496606 1437 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 3 LEADER]: Stepping down as leader of term 3
I20260501 08:10:51.496749 1437 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:51.496858 1437 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.497069 1437 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 3 FOLLOWER]: Advancing to term 4
I20260501 08:10:51.595438 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 4 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20260501 08:10:51.595782 1438 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20260501 08:10:51.596282 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 4 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20260501 08:10:51.596562 1438 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20260501 08:10:51.794572 1483 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:51.794693 1483 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 4 FOLLOWER]: Advancing to term 5
I20260501 08:10:51.795519 1483 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.795715 1483 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 5 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:51.796000 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:51.796025 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:51.796368 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:51.796473 1483 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260501 08:10:51.797986 1234 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 5 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 4. Current term is 5. Ops: []
I20260501 08:10:51.798219 1484 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 4.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20260501 08:10:51.825466 1484 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 4 LEADER]: Stepping down as leader of term 4
I20260501 08:10:51.825608 1484 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:51.825709 1484 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.825842 1484 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 4 FOLLOWER]: Advancing to term 5
I20260501 08:10:51.861331 1437 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:51.861461 1437 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 4 FOLLOWER]: Advancing to term 5
I20260501 08:10:51.862303 1437 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.862581 1437 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 5 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:51.862834 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:51.862850 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:51.862979 1234 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 5: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:10:51.863665 1385 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 5.
I20260501 08:10:51.863940 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:51.864082 1437 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 5 FOLLOWER]: Leader election won for term 5
I20260501 08:10:51.864220 1437 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 5 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:51.864363 1437 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.864874 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 4 to 5, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 5 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:51.898548 1483 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:51.898669 1483 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 5 FOLLOWER]: Advancing to term 6
I20260501 08:10:51.899400 1483 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.899639 1483 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 6 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:51.899837 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 6 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:51.899941 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 6 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:51.900066 1385 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 5 FOLLOWER]: Advancing to term 6
I20260501 08:10:51.900653 1385 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 6.
I20260501 08:10:51.900863 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:51.901021 1483 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 6 FOLLOWER]: Leader election won for term 6
I20260501 08:10:51.901124 1483 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 6 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:51.901180 1483 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.950397 1385 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 6 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 5. Current term is 6. Ops: []
I20260501 08:10:51.950748 1437 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20260501 08:10:51.954869 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 5 to 6, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 6 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:51.990070 1234 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 6 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 5. Current term is 6. Ops: []
I20260501 08:10:51.990468 1481 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20260501 08:10:51.990623 1481 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 6.43 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 42
I20260501 08:10:51.991006 1385 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 6 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 6 index: 43. (index mismatch)
I20260501 08:10:51.991398 1483 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20260501 08:10:51.994024 1437 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 5 LEADER]: Stepping down as leader of term 5
I20260501 08:10:51.994096 1437 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:51.994179 1437 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.43, Last appended by leader: 43, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:51.994278 1437 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 5 FOLLOWER]: Advancing to term 6
I20260501 08:10:52.000159 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 6 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 6 index: 43. (term mismatch)
I20260501 08:10:52.000272 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 42
I20260501 08:10:52.000296 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 5.43
I20260501 08:10:52.000320 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:52.000588 1483 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20260501 08:10:52.341327 1484 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 6 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:52.341434 1484 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 6 FOLLOWER]: Advancing to term 7
I20260501 08:10:52.342104 1484 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.342303 1484 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 7 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:52.342545 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:52.342568 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:52.342836 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:52.343039 1484 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260501 08:10:52.352650 1385 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 7 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 6. Current term is 7. Ops: []
I20260501 08:10:52.352926 1480 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 6.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20260501 08:10:52.393303 1437 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 6 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:52.393415 1437 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 6 FOLLOWER]: Advancing to term 7
I20260501 08:10:52.394281 1437 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.394480 1437 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 7 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:52.394730 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:52.394730 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:52.394866 1385 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 7: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:10:52.395044 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:52.395182 1437 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260501 08:10:52.408967 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 7 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 6. Current term is 7. Ops: []
I20260501 08:10:52.409327 1480 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 6.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20260501 08:10:52.421221 1486 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 6 LEADER]: Stepping down as leader of term 6
I20260501 08:10:52.421316 1486 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:52.421422 1486 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 6.43, Last appended by leader: 43, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.421532 1486 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 6 FOLLOWER]: Advancing to term 7
I20260501 08:10:52.445192 1484 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:52.445315 1484 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 7 FOLLOWER]: Advancing to term 8
I20260501 08:10:52.446000 1484 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.446364 1484 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 8 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:52.446548 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 8 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:52.446630 1234 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 7 FOLLOWER]: Advancing to term 8
I20260501 08:10:52.447459 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 8.
I20260501 08:10:52.447579 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 8 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:52.447638 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 7 FOLLOWER]: Advancing to term 8
I20260501 08:10:52.447683 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters:
I20260501 08:10:52.447829 1484 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 8 FOLLOWER]: Leader election won for term 8
I20260501 08:10:52.448064 1484 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 8 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:52.448266 1484 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 6.43, Last appended by leader: 43, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.448660 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 8.
I20260501 08:10:52.448958 1129 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 6 to 8, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 8 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:52.550009 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 8 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 6 index: 43. Preceding OpId from leader: term: 8 index: 44. (index mismatch)
I20260501 08:10:52.550472 1484 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20260501 08:10:52.568550 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 8 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 6 index: 43. Preceding OpId from leader: term: 8 index: 44. (index mismatch)
I20260501 08:10:52.568883 1487 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20260501 08:10:52.652145 1437 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:52.652253 1437 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 8 FOLLOWER]: Advancing to term 9
I20260501 08:10:52.652902 1437 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.653184 1437 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 9 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:52.653331 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 9 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 8. Current term is 9. Ops: []
I20260501 08:10:52.653288 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 9 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:52.653373 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 9 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:52.653563 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:52.653657 1437 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260501 08:10:52.653687 1488 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 8.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20260501 08:10:52.670356 1486 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:52.670467 1486 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 8 FOLLOWER]: Advancing to term 9
I20260501 08:10:52.671180 1486 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.671381 1486 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 9 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:52.671627 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 9 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:52.671627 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 9 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:52.671768 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 9: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:52.672149 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:52.672255 1486 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260501 08:10:52.685580 1234 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 9 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 8. Current term is 9. Ops: []
I20260501 08:10:52.685911 1484 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 8.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20260501 08:10:52.757225 1487 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 8 LEADER]: Stepping down as leader of term 8
I20260501 08:10:52.757356 1487 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:52.757440 1487 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 8.44, Last appended by leader: 44, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.757597 1487 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 8 FOLLOWER]: Advancing to term 9
I20260501 08:10:52.799911 1486 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:52.800035 1486 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 9 FOLLOWER]: Advancing to term 10
I20260501 08:10:52.800766 1486 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.800959 1486 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 10 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:52.801167 1437 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:52.801203 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 10 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:52.801232 1437 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 9 FOLLOWER]: Advancing to term 10
I20260501 08:10:52.801342 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 10 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:52.801462 1385 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 9 FOLLOWER]: Advancing to term 10
I20260501 08:10:52.801874 1437 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.801994 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 10: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:52.802116 1385 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 10.
I20260501 08:10:52.802220 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 10 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:52.802243 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 10 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:52.802275 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:52.802304 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 10: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:10:52.802311 1234 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 10: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:10:52.802377 1486 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 10 FOLLOWER]: Leader election won for term 10
I20260501 08:10:52.802134 1437 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 10 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:52.802497 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:52.802424 1486 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 10 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:52.802570 1437 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260501 08:10:52.802611 1486 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 8.44, Last appended by leader: 44, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.803103 1129 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 8 to 10, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 10 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:52.881876 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 10 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 8 index: 44. Preceding OpId from leader: term: 10 index: 45. (index mismatch)
I20260501 08:10:52.882273 1486 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20260501 08:10:52.885635 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 10 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 8 index: 44. Preceding OpId from leader: term: 10 index: 45. (index mismatch)
I20260501 08:10:52.886008 1480 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20260501 08:10:52.984443 1487 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 10 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:52.984555 1487 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 10 FOLLOWER]: Advancing to term 11
I20260501 08:10:52.985317 1487 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:52.985513 1487 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 11 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:52.985739 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 11 candidate_status { last_received { term: 10 index: 45 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:52.985780 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 11 candidate_status { last_received { term: 10 index: 45 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:52.986074 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:52.986193 1487 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260501 08:10:53.007716 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 11 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 10. Current term is 11. Ops: []
I20260501 08:10:53.008103 1486 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 10.45, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20260501 08:10:53.043386 1486 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 10 LEADER]: Stepping down as leader of term 10
I20260501 08:10:53.043569 1486 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 10 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:53.043684 1486 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 10.45, Last appended by leader: 45, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.043797 1486 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 10 FOLLOWER]: Advancing to term 11
I20260501 08:10:53.047304 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:53.048604 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:53.050843 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:53.055208 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:53.063441 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:53.068622 1437 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 10 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:53.068729 1437 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 10 FOLLOWER]: Advancing to term 11
I20260501 08:10:53.069525 1437 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.069752 1437 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 11 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:53.069972 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 11 candidate_status { last_received { term: 10 index: 45 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:53.070019 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 11 candidate_status { last_received { term: 10 index: 45 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:53.070097 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 11: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:10:53.070689 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 11.
I20260501 08:10:53.070971 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters:
I20260501 08:10:53.071159 1437 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 11 FOLLOWER]: Leader election won for term 11
I20260501 08:10:53.071305 1437 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 11 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:53.071388 1437 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 10.45, Last appended by leader: 45, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.071990 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 10 to 11, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 11 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:53.079711 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:53.112118 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:53.114084 1487 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:53.114307 1487 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 11 FOLLOWER]: Advancing to term 12
I20260501 08:10:53.115010 1487 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.115223 1487 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 12 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:53.115449 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 12 candidate_status { last_received { term: 10 index: 45 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:53.115464 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 12 candidate_status { last_received { term: 10 index: 45 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:53.115561 1234 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 11 FOLLOWER]: Advancing to term 12
I20260501 08:10:53.116279 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 12.
I20260501 08:10:53.116524 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:53.116652 1487 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 12 FOLLOWER]: Leader election won for term 12
I20260501 08:10:53.116726 1487 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 12 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:53.116803 1487 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 10.45, Last appended by leader: 45, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.117435 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 11 to 12, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 12 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:53.169838 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 12 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 11. Current term is 12. Ops: []
I20260501 08:10:53.170231 1437 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20260501 08:10:53.170346 1437 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 12.46 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 45
I20260501 08:10:53.170370 1437 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 11 LEADER]: Stepping down as leader of term 11
I20260501 08:10:53.170385 1437 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:53.170480 1437 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 11.46, Last appended by leader: 46, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.170554 1437 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 11 FOLLOWER]: Advancing to term 12
I20260501 08:10:53.176431 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:53.196817 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 12 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 10 index: 45. Preceding OpId from leader: term: 12 index: 46. (index mismatch)
I20260501 08:10:53.197197 1487 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20260501 08:10:53.198416 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 12 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 11 index: 46. Preceding OpId from leader: term: 12 index: 46. (term mismatch)
I20260501 08:10:53.198477 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 45
I20260501 08:10:53.198501 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 11.46
I20260501 08:10:53.198520 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:53.198678 1487 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20260501 08:10:53.298789 1486 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 12 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:53.298926 1486 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 12 FOLLOWER]: Advancing to term 13
I20260501 08:10:53.299787 1486 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.300022 1486 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 13 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:53.300273 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 13 candidate_status { last_received { term: 12 index: 46 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:53.300288 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 13 candidate_status { last_received { term: 12 index: 46 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:53.300423 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 12 FOLLOWER]: Advancing to term 13
I20260501 08:10:53.301025 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 13.
I20260501 08:10:53.301298 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:53.301417 1486 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 13 FOLLOWER]: Leader election won for term 13
I20260501 08:10:53.301502 1486 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 13 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:53.301553 1486 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 12.46, Last appended by leader: 46, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.302098 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 12 to 13, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 13 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:53.304661 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:53.315135 1234 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 13 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 12. Current term is 13. Ops: []
I20260501 08:10:53.315479 1487 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 12.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20260501 08:10:53.315577 1487 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 13.47 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 46
I20260501 08:10:53.315600 1487 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 12 LEADER]: Stepping down as leader of term 12
I20260501 08:10:53.315613 1487 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:53.315680 1487 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 12.46, Last appended by leader: 46, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.315758 1487 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 12 FOLLOWER]: Advancing to term 13
I20260501 08:10:53.400678 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 13 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 12 index: 46. Preceding OpId from leader: term: 13 index: 47. (index mismatch)
I20260501 08:10:53.401216 1486 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20260501 08:10:53.402352 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 13 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 12 index: 46. Preceding OpId from leader: term: 13 index: 47. (index mismatch)
I20260501 08:10:53.402544 1486 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20260501 08:10:53.503595 1489 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 13 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:53.503706 1489 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 13 FOLLOWER]: Advancing to term 14
I20260501 08:10:53.504468 1489 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.504675 1489 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 14 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:53.504943 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 14 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:53.504910 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 14 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:53.505251 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:53.505353 1489 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260501 08:10:53.514827 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 14 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 13. Current term is 14. Ops: []
I20260501 08:10:53.515249 1480 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 13.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260501 08:10:53.515372 1480 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 13 LEADER]: Stepping down as leader of term 13
I20260501 08:10:53.515389 1480 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 13 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:53.515455 1480 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 13.47, Last appended by leader: 47, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.515542 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 13 FOLLOWER]: Advancing to term 14
I20260501 08:10:53.560967 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:53.582441 1487 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 13 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:53.582562 1487 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 13 FOLLOWER]: Advancing to term 14
I20260501 08:10:53.583241 1487 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.583469 1487 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 14 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:53.583662 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 14 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:53.583755 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 14 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:53.583794 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 14: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:53.584498 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 14.
I20260501 08:10:53.584719 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:53.584972 1487 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 14 FOLLOWER]: Leader election won for term 14
I20260501 08:10:53.585062 1487 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 14 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:53.585110 1487 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 13.47, Last appended by leader: 47, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.585640 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 13 to 14, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 14 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:53.609516 1489 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:53.609622 1489 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 14 FOLLOWER]: Advancing to term 15
I20260501 08:10:53.610293 1489 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.610500 1489 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 15 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:53.610706 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 15 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:53.610706 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 15 candidate_status { last_received { term: 13 index: 47 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:53.610836 1234 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 14 FOLLOWER]: Advancing to term 15
I20260501 08:10:53.611434 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 15.
I20260501 08:10:53.611615 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:53.611757 1489 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 15 FOLLOWER]: Leader election won for term 15
I20260501 08:10:53.611832 1489 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 15 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:53.611901 1489 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 13.47, Last appended by leader: 47, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.612474 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 14 to 15, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 15 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:53.670063 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 15 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 14. Current term is 15. Ops: []
I20260501 08:10:53.670406 1487 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260501 08:10:53.670544 1487 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 15.48 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 47
I20260501 08:10:53.670583 1488 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 14 LEADER]: Stepping down as leader of term 14
I20260501 08:10:53.670620 1488 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:53.670706 1488 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 14.48, Last appended by leader: 48, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.670786 1488 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 14 FOLLOWER]: Advancing to term 15
I20260501 08:10:53.701159 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 15 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 14 index: 48. Preceding OpId from leader: term: 15 index: 48. (term mismatch)
I20260501 08:10:53.701267 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 47
I20260501 08:10:53.701283 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 14.48
I20260501 08:10:53.701300 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:53.701658 1489 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260501 08:10:53.703032 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 15 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 13 index: 47. Preceding OpId from leader: term: 15 index: 48. (index mismatch)
I20260501 08:10:53.703225 1437 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260501 08:10:53.896306 1488 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:10:53.896414 1488 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 15 FOLLOWER]: Advancing to term 16
I20260501 08:10:53.897166 1488 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.897375 1488 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 16 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:53.897607 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 16 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:53.897629 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 16 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:53.897924 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:53.898069 1488 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260501 08:10:53.914961 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 16 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 15. Current term is 16. Ops: []
I20260501 08:10:53.915314 1490 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 15.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20260501 08:10:53.915458 1490 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 15 LEADER]: Stepping down as leader of term 15
I20260501 08:10:53.915508 1490 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:53.915575 1490 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 15.48, Last appended by leader: 48, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.915653 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 15 FOLLOWER]: Advancing to term 16
I20260501 08:10:53.980376 1480 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:10:53.980521 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 15 FOLLOWER]: Advancing to term 16
I20260501 08:10:53.981240 1480 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.981472 1480 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 16 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:53.981714 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 16 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:53.981720 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 16 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:53.981863 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 16: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:10:53.982432 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 16.
I20260501 08:10:53.982753 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:53.982877 1480 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 16 FOLLOWER]: Leader election won for term 16
I20260501 08:10:53.982957 1480 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 16 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:53.983057 1480 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 15.48, Last appended by leader: 48, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:53.983577 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 15 to 16, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 16 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:54.014472 1488 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:54.014633 1488 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 16 FOLLOWER]: Advancing to term 17
I20260501 08:10:54.015400 1488 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.015584 1488 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 17 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:54.015769 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 17 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:54.015815 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 17 candidate_status { last_received { term: 15 index: 48 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:54.015829 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 16 FOLLOWER]: Advancing to term 17
I20260501 08:10:54.016615 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 17.
I20260501 08:10:54.016845 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:54.016978 1488 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 17 FOLLOWER]: Leader election won for term 17
I20260501 08:10:54.017056 1488 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 17 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:54.017135 1488 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 15.48, Last appended by leader: 48, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.017678 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 16 to 17, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 17 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:54.068116 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 17 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 16. Current term is 17. Ops: []
I20260501 08:10:54.068588 1480 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20260501 08:10:54.068711 1480 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 16 LEADER]: Stepping down as leader of term 16
I20260501 08:10:54.068729 1480 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 16 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:54.068811 1480 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 16.49, Last appended by leader: 49, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.068908 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 16 FOLLOWER]: Advancing to term 17
I20260501 08:10:54.073261 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:54.108374 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 17 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 16 index: 49. Preceding OpId from leader: term: 17 index: 49. (term mismatch)
I20260501 08:10:54.108448 1234 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 48
I20260501 08:10:54.108461 1234 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 16.49
I20260501 08:10:54.108476 1234 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:54.108790 1488 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20260501 08:10:54.110056 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 17 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 15 index: 48. Preceding OpId from leader: term: 17 index: 49. (index mismatch)
I20260501 08:10:54.110234 1491 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20260501 08:10:54.211208 1490 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:54.211308 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 17 FOLLOWER]: Advancing to term 18
I20260501 08:10:54.211972 1490 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.212210 1490 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 18 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:54.212371 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 18 candidate_status { last_received { term: 17 index: 49 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:54.212425 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 18 candidate_status { last_received { term: 17 index: 49 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:54.212679 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:54.212775 1490 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260501 08:10:54.221218 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 18 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 17. Current term is 18. Ops: []
I20260501 08:10:54.221501 1491 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 17.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20260501 08:10:54.221648 1491 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 17 LEADER]: Stepping down as leader of term 17
I20260501 08:10:54.221688 1491 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:54.221752 1491 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 17.49, Last appended by leader: 49, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.221853 1491 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 17 FOLLOWER]: Advancing to term 18
I20260501 08:10:54.296276 1480 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 17 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:54.296424 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 17 FOLLOWER]: Advancing to term 18
I20260501 08:10:54.297148 1480 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.297395 1480 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 18 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:54.297629 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 18 candidate_status { last_received { term: 17 index: 49 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:54.297767 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 18: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:54.297629 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 18 candidate_status { last_received { term: 17 index: 49 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:54.298524 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 18.
I20260501 08:10:54.298744 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:54.298897 1480 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 18 FOLLOWER]: Leader election won for term 18
I20260501 08:10:54.298981 1480 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 18 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:54.299058 1480 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 17.49, Last appended by leader: 49, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.299574 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 17 to 18, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 18 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:54.340346 1490 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:54.340451 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 18 FOLLOWER]: Advancing to term 19
I20260501 08:10:54.341179 1490 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.341378 1490 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 19 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:54.341624 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 19 candidate_status { last_received { term: 17 index: 49 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:54.341624 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 19 candidate_status { last_received { term: 17 index: 49 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:54.341732 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 18 FOLLOWER]: Advancing to term 19
I20260501 08:10:54.342357 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 19.
I20260501 08:10:54.342657 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:54.342747 1490 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 19 FOLLOWER]: Leader election won for term 19
I20260501 08:10:54.342993 1490 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 19 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:54.343087 1490 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 17.49, Last appended by leader: 49, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.343582 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 18 to 19, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 19 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:54.417078 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 19 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 18. Current term is 19. Ops: []
I20260501 08:10:54.417537 1480 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20260501 08:10:54.417773 1480 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 18 LEADER]: Stepping down as leader of term 18
I20260501 08:10:54.417815 1480 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 18 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:54.417891 1480 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 18.50, Last appended by leader: 50, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.417973 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 18 FOLLOWER]: Advancing to term 19
I20260501 08:10:54.437034 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 19 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 18 index: 50. Preceding OpId from leader: term: 19 index: 50. (term mismatch)
I20260501 08:10:54.437114 1234 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 49
I20260501 08:10:54.437129 1234 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 18.50
I20260501 08:10:54.437143 1234 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:54.437388 1490 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20260501 08:10:54.438684 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 19 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 17 index: 49. Preceding OpId from leader: term: 19 index: 50. (index mismatch)
I20260501 08:10:54.438874 1492 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20260501 08:10:54.623917 1491 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:10:54.624032 1491 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 19 FOLLOWER]: Advancing to term 20
I20260501 08:10:54.624782 1491 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.624961 1491 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 20 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:54.625216 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 20 candidate_status { last_received { term: 19 index: 50 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:54.625218 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 20 candidate_status { last_received { term: 19 index: 50 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:54.625512 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:54.625628 1491 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260501 08:10:54.638659 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 20 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 19. Current term is 20. Ops: []
I20260501 08:10:54.639006 1490 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 19.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20260501 08:10:54.639149 1492 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 19 LEADER]: Stepping down as leader of term 19
I20260501 08:10:54.639195 1492 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:54.639261 1492 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 19.50, Last appended by leader: 50, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.639329 1492 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 19 FOLLOWER]: Advancing to term 20
I20260501 08:10:54.643430 1480 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:10:54.643517 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 19 FOLLOWER]: Advancing to term 20
I20260501 08:10:54.644207 1480 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.644375 1480 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 20 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:54.644579 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 20 candidate_status { last_received { term: 19 index: 50 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:54.644577 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 20 candidate_status { last_received { term: 19 index: 50 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:54.644693 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 20: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:10:54.645401 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 20.
I20260501 08:10:54.645570 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:54.645699 1480 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 20 FOLLOWER]: Leader election won for term 20
I20260501 08:10:54.645789 1480 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 20 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:54.645838 1480 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 19.50, Last appended by leader: 50, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.646387 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 19 to 20, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 20 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:54.746063 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 20 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 19 index: 50. Preceding OpId from leader: term: 20 index: 51. (index mismatch)
I20260501 08:10:54.746438 1480 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20260501 08:10:54.747928 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 20 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 19 index: 50. Preceding OpId from leader: term: 20 index: 51. (index mismatch)
I20260501 08:10:54.748145 1483 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20260501 08:10:54.940605 1491 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 20 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:54.940711 1491 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 20 FOLLOWER]: Advancing to term 21
I20260501 08:10:54.941447 1491 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.941646 1491 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 21 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:54.941921 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 21 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:54.941942 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 21 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:54.942270 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:54.942384 1491 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260501 08:10:54.947439 1490 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 20 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:54.947535 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 20 FOLLOWER]: Advancing to term 21
I20260501 08:10:54.948257 1490 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.948468 1490 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 21 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:54.948694 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 21 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:54.948683 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 21 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:54.948853 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 21: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:10:54.949051 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:54.949139 1490 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260501 08:10:54.949549 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 21 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 20. Current term is 21. Ops: []
I20260501 08:10:54.949806 1493 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 20.51, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20260501 08:10:54.950176 1493 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 20 LEADER]: Stepping down as leader of term 20
I20260501 08:10:54.950244 1493 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 20 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:54.950312 1493 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 20.51, Last appended by leader: 51, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:54.950424 1493 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 20 FOLLOWER]: Advancing to term 21
I20260501 08:10:55.073618 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:55.075031 1491 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:55.075112 1491 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 21 FOLLOWER]: Advancing to term 22
I20260501 08:10:55.075740 1491 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.075991 1491 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 22 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:55.076179 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 22 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:55.076259 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 21 FOLLOWER]: Advancing to term 22
I20260501 08:10:55.076279 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 22 candidate_status { last_received { term: 20 index: 51 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:55.076471 1234 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 21 FOLLOWER]: Advancing to term 22
I20260501 08:10:55.076900 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 22.
I20260501 08:10:55.077118 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters:
I20260501 08:10:55.077196 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 22.
I20260501 08:10:55.077239 1491 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 22 FOLLOWER]: Leader election won for term 22
I20260501 08:10:55.077368 1491 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 22 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:55.077416 1491 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 20.51, Last appended by leader: 51, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.077869 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 20 to 22, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 22 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:55.157357 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 22 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 20 index: 51. Preceding OpId from leader: term: 22 index: 52. (index mismatch)
I20260501 08:10:55.157644 1491 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20260501 08:10:55.158746 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 22 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 20 index: 51. Preceding OpId from leader: term: 22 index: 52. (index mismatch)
I20260501 08:10:55.159029 1491 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20260501 08:10:55.259958 1493 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 22 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:55.260182 1493 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 22 FOLLOWER]: Advancing to term 23
I20260501 08:10:55.260924 1493 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.261171 1493 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 23 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:55.261397 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 23 candidate_status { last_received { term: 22 index: 52 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:55.261384 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 23 candidate_status { last_received { term: 22 index: 52 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:55.261670 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:55.261801 1493 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260501 08:10:55.262574 1234 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 23 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 22. Current term is 23. Ops: []
I20260501 08:10:55.262814 1495 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 22.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20260501 08:10:55.262939 1495 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 22 LEADER]: Stepping down as leader of term 22
I20260501 08:10:55.262969 1495 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:55.263048 1495 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 22.52, Last appended by leader: 52, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.263144 1495 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 22 FOLLOWER]: Advancing to term 23
I20260501 08:10:55.359856 1490 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:55.360018 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 22 FOLLOWER]: Advancing to term 23
I20260501 08:10:55.360688 1490 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.360893 1490 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 23 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:55.361100 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 23 candidate_status { last_received { term: 22 index: 52 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:55.361127 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 23 candidate_status { last_received { term: 22 index: 52 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:55.361250 1234 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 23: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:10:55.361811 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 23.
I20260501 08:10:55.362025 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:55.362118 1490 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 23 FOLLOWER]: Leader election won for term 23
I20260501 08:10:55.362242 1490 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 23 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:55.362300 1490 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 22.52, Last appended by leader: 52, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.362870 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 22 to 23, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 23 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:55.409325 1493 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:55.409431 1493 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 23 FOLLOWER]: Advancing to term 24
I20260501 08:10:55.410207 1493 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.410450 1493 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 24 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:55.410681 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 24 candidate_status { last_received { term: 22 index: 52 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:55.410642 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 24 candidate_status { last_received { term: 22 index: 52 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:55.410759 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 23 FOLLOWER]: Advancing to term 24
I20260501 08:10:55.411330 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 24.
I20260501 08:10:55.411540 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:55.411641 1493 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 24 FOLLOWER]: Leader election won for term 24
I20260501 08:10:55.411705 1493 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 24 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:55.411763 1493 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 22.52, Last appended by leader: 52, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.412590 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 23 to 24, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 24 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:55.471292 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 24 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 23. Current term is 24. Ops: []
I20260501 08:10:55.471625 1490 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20260501 08:10:55.471769 1490 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 23 LEADER]: Stepping down as leader of term 23
I20260501 08:10:55.471807 1490 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:55.471868 1490 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 23.53, Last appended by leader: 53, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.471951 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 23 FOLLOWER]: Advancing to term 24
I20260501 08:10:55.494840 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 24 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 22 index: 52. Preceding OpId from leader: term: 24 index: 53. (index mismatch)
I20260501 08:10:55.495218 1493 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20260501 08:10:55.496447 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 24 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 23 index: 53. Preceding OpId from leader: term: 24 index: 53. (term mismatch)
I20260501 08:10:55.496503 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 52
I20260501 08:10:55.496543 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 23.53
I20260501 08:10:55.496598 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:55.496810 1493 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20260501 08:10:55.720599 1490 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 24 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:55.720703 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 24 FOLLOWER]: Advancing to term 25
I20260501 08:10:55.721371 1490 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.721567 1490 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 25 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:55.721801 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 25 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:55.721801 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 25 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:55.722106 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:55.722208 1490 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260501 08:10:55.725476 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 25 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 24. Current term is 25. Ops: []
I20260501 08:10:55.725809 1497 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 24.53, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20260501 08:10:55.725955 1497 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 24 LEADER]: Stepping down as leader of term 24
I20260501 08:10:55.726017 1497 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 24 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:55.726083 1497 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 24.53, Last appended by leader: 53, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.726217 1497 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 24 FOLLOWER]: Advancing to term 25
I20260501 08:10:55.773471 1495 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 24 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:55.773577 1495 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 24 FOLLOWER]: Advancing to term 25
I20260501 08:10:55.774390 1495 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.774585 1495 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 25 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:55.774809 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 25 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:55.774809 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 25 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:55.775030 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 25: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:55.775964 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 25.
I20260501 08:10:55.776538 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:55.776816 1495 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 25 FOLLOWER]: Leader election won for term 25
I20260501 08:10:55.777099 1495 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 25 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:55.777256 1495 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 24.53, Last appended by leader: 53, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.777798 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 24 to 25, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 25 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:55.835738 1490 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:55.835871 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 25 FOLLOWER]: Advancing to term 26
I20260501 08:10:55.836774 1490 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.837167 1490 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 26 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:55.837262 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 26 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:55.837325 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 26 candidate_status { last_received { term: 24 index: 53 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:55.837417 1234 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 25 FOLLOWER]: Advancing to term 26
I20260501 08:10:55.838068 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 26.
I20260501 08:10:55.838327 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:55.838472 1490 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 26 FOLLOWER]: Leader election won for term 26
I20260501 08:10:55.838551 1490 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 26 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:55.838610 1490 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 24.53, Last appended by leader: 53, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.839174 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 25 to 26, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 26 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:55.872298 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 26 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 25. Current term is 26. Ops: []
I20260501 08:10:55.872649 1495 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20260501 08:10:55.872753 1495 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 26.54 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 53
I20260501 08:10:55.872777 1495 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 25 LEADER]: Stepping down as leader of term 25
I20260501 08:10:55.872792 1495 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:55.872857 1495 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 25.54, Last appended by leader: 54, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:55.872933 1495 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 25 FOLLOWER]: Advancing to term 26
I20260501 08:10:55.929579 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 26 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 25 index: 54. Preceding OpId from leader: term: 26 index: 54. (term mismatch)
I20260501 08:10:55.929665 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 53
I20260501 08:10:55.929679 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 25.54
I20260501 08:10:55.929693 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:55.929994 1490 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20260501 08:10:55.931305 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 26 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 24 index: 53. Preceding OpId from leader: term: 26 index: 54. (index mismatch)
I20260501 08:10:55.931587 1490 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20260501 08:10:56.032704 1497 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:10:56.032835 1497 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 26 FOLLOWER]: Advancing to term 27
I20260501 08:10:56.033525 1497 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.033739 1497 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 27 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:56.033978 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 27 candidate_status { last_received { term: 26 index: 54 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:56.033995 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 27 candidate_status { last_received { term: 26 index: 54 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:56.034261 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:56.034426 1497 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260501 08:10:56.036716 1234 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 27 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 26. Current term is 27. Ops: []
I20260501 08:10:56.036980 1490 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 26.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260501 08:10:56.037107 1490 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 26 LEADER]: Stepping down as leader of term 26
I20260501 08:10:56.037150 1490 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:56.037216 1490 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 26.54, Last appended by leader: 54, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.037302 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 26 FOLLOWER]: Advancing to term 27
I20260501 08:10:56.074008 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:56.127641 1495 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:10:56.127743 1495 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 26 FOLLOWER]: Advancing to term 27
I20260501 08:10:56.128548 1495 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.128746 1495 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 27 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:56.128999 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 27 candidate_status { last_received { term: 26 index: 54 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:56.129005 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 27 candidate_status { last_received { term: 26 index: 54 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:56.129127 1234 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 27: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:10:56.129776 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 27.
I20260501 08:10:56.130043 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:56.130151 1495 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 27 FOLLOWER]: Leader election won for term 27
I20260501 08:10:56.130198 1495 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 27 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:56.130255 1495 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 26.54, Last appended by leader: 54, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.130805 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 26 to 27, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 27 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:56.160055 1497 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:56.160184 1497 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 27 FOLLOWER]: Advancing to term 28
I20260501 08:10:56.160863 1497 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.161103 1497 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 28 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:56.161314 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 28 candidate_status { last_received { term: 26 index: 54 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:56.161309 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 28 candidate_status { last_received { term: 26 index: 54 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:56.161504 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 27 FOLLOWER]: Advancing to term 28
I20260501 08:10:56.162111 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 28.
I20260501 08:10:56.162340 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:56.162484 1497 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 28 FOLLOWER]: Leader election won for term 28
I20260501 08:10:56.162580 1497 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 28 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:56.162629 1497 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 26.54, Last appended by leader: 54, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.163255 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 27 to 28, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 28 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:56.213243 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 28 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 27. Current term is 28. Ops: []
I20260501 08:10:56.213642 1495 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260501 08:10:56.213778 1495 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 27 LEADER]: Stepping down as leader of term 27
I20260501 08:10:56.213798 1495 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:56.213872 1495 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 27.55, Last appended by leader: 55, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.213951 1495 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 27 FOLLOWER]: Advancing to term 28
I20260501 08:10:56.267781 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 28 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 26 index: 54. Preceding OpId from leader: term: 28 index: 55. (index mismatch)
I20260501 08:10:56.268149 1497 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260501 08:10:56.269560 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 28 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 27 index: 55. Preceding OpId from leader: term: 28 index: 55. (term mismatch)
I20260501 08:10:56.269620 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 54
I20260501 08:10:56.269644 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 27.55
I20260501 08:10:56.269671 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:56.269835 1500 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20260501 08:10:56.465663 1490 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 28 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:56.465766 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 28 FOLLOWER]: Advancing to term 29
I20260501 08:10:56.466450 1490 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.466655 1490 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 29 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:56.466884 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 29 candidate_status { last_received { term: 28 index: 55 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:56.466996 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 29 candidate_status { last_received { term: 28 index: 55 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:56.467272 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:56.467396 1490 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260501 08:10:56.474020 1495 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 28 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:56.474117 1495 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 28 FOLLOWER]: Advancing to term 29
I20260501 08:10:56.475812 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 29 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 28. Current term is 29. Ops: []
I20260501 08:10:56.475803 1495 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.476178 1495 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 29 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:56.476258 1500 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 28.55, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20260501 08:10:56.476298 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 29 candidate_status { last_received { term: 28 index: 55 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:56.476372 1480 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 28 LEADER]: Stepping down as leader of term 28
I20260501 08:10:56.476413 1480 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 28 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:56.476377 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 29: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:56.476383 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 29 candidate_status { last_received { term: 28 index: 55 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:56.476477 1480 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 28.55, Last appended by leader: 55, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.476558 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 28 FOLLOWER]: Advancing to term 29
I20260501 08:10:56.477798 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 29.
I20260501 08:10:56.478013 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:56.478106 1495 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 29 FOLLOWER]: Leader election won for term 29
I20260501 08:10:56.478159 1495 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 29 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:56.478242 1495 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 28.55, Last appended by leader: 55, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.478688 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 28 to 29, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 29 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:56.569864 1490 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:56.569974 1490 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 29 FOLLOWER]: Advancing to term 30
I20260501 08:10:56.570665 1490 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.570866 1490 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 30 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:56.571105 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 30 candidate_status { last_received { term: 28 index: 55 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:56.571195 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 30 candidate_status { last_received { term: 28 index: 55 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:56.571265 1234 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 29 FOLLOWER]: Advancing to term 30
I20260501 08:10:56.571851 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 30.
I20260501 08:10:56.572167 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:56.572302 1490 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 30 FOLLOWER]: Leader election won for term 30
I20260501 08:10:56.572433 1490 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 30 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:56.572479 1490 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 28.55, Last appended by leader: 55, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.573005 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 29 to 30, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 30 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:56.589300 1234 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 30 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 29. Current term is 30. Ops: []
I20260501 08:10:56.589622 1495 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20260501 08:10:56.589766 1495 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 29 LEADER]: Stepping down as leader of term 29
I20260501 08:10:56.589805 1495 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:56.589874 1495 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 29.56, Last appended by leader: 56, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.589991 1495 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 29 FOLLOWER]: Advancing to term 30
I20260501 08:10:56.665550 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 30 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 28 index: 55. Preceding OpId from leader: term: 30 index: 56. (index mismatch)
I20260501 08:10:56.665870 1490 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20260501 08:10:56.667160 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 30 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 29 index: 56. Preceding OpId from leader: term: 30 index: 56. (term mismatch)
I20260501 08:10:56.667219 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 55
I20260501 08:10:56.667239 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 29.56
I20260501 08:10:56.667258 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:56.667409 1502 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20260501 08:10:56.767990 1480 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:10:56.768100 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 30 FOLLOWER]: Advancing to term 31
I20260501 08:10:56.768854 1480 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.769060 1480 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 31 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:56.769356 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 31 candidate_status { last_received { term: 30 index: 56 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:56.769356 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 31 candidate_status { last_received { term: 30 index: 56 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:56.769583 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 30 FOLLOWER]: Advancing to term 31
I20260501 08:10:56.770285 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 31.
I20260501 08:10:56.770524 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:56.770629 1480 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 31 FOLLOWER]: Leader election won for term 31
I20260501 08:10:56.770699 1480 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 31 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:56.770754 1480 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 30.56, Last appended by leader: 56, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.771377 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 30 to 31, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 31 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:56.777292 1234 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 31 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 30. Current term is 31. Ops: []
I20260501 08:10:56.777542 1502 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 30.56, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20260501 08:10:56.777693 1502 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 31.57 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 56
I20260501 08:10:56.777726 1501 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 30 LEADER]: Stepping down as leader of term 30
I20260501 08:10:56.777768 1501 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:56.777837 1501 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 30.56, Last appended by leader: 56, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:56.777973 1501 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 30 FOLLOWER]: Advancing to term 31
I20260501 08:10:56.871459 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 31 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 30 index: 56. Preceding OpId from leader: term: 31 index: 57. (index mismatch)
I20260501 08:10:56.871814 1480 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20260501 08:10:56.873000 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 31 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 30 index: 56. Preceding OpId from leader: term: 31 index: 57. (index mismatch)
I20260501 08:10:56.873342 1497 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20260501 08:10:57.074354 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:57.091061 1502 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 31 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:57.091199 1502 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 31 FOLLOWER]: Advancing to term 32
I20260501 08:10:57.092000 1502 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.092224 1502 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 32 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:57.092463 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 32 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:57.092463 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 32 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:57.092725 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:57.092830 1502 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260501 08:10:57.106388 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 32 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 31. Current term is 32. Ops: []
I20260501 08:10:57.106724 1480 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 31.57, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20260501 08:10:57.106900 1480 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 31 LEADER]: Stepping down as leader of term 31
I20260501 08:10:57.106978 1480 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 31 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:57.107077 1480 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 31.57, Last appended by leader: 57, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.107167 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 31 FOLLOWER]: Advancing to term 32
I20260501 08:10:57.132381 1495 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 31 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:57.132480 1495 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 31 FOLLOWER]: Advancing to term 32
I20260501 08:10:57.133311 1495 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.133507 1495 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 32 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:57.133769 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 32 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:57.133775 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 32 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:57.133865 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 32: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:57.134528 1234 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 32.
I20260501 08:10:57.134754 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:57.134855 1495 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 32 FOLLOWER]: Leader election won for term 32
I20260501 08:10:57.134915 1495 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 32 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:57.135017 1495 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 31.57, Last appended by leader: 57, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.135550 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 31 to 32, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 32 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:57.215298 1502 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:57.215415 1502 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 32 FOLLOWER]: Advancing to term 33
I20260501 08:10:57.216092 1234 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 32 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 31 index: 57. Preceding OpId from leader: term: 32 index: 58. (index mismatch)
I20260501 08:10:57.216408 1502 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.216418 1495 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20260501 08:10:57.216594 1502 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 33 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:57.216737 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 33 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:57.216791 1234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 33 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:57.216995 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:57.217118 1502 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260501 08:10:57.217657 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 33 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 32. Current term is 33. Ops: []
I20260501 08:10:57.217919 1498 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20260501 08:10:57.218034 1498 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 32 LEADER]: Stepping down as leader of term 32
I20260501 08:10:57.218070 1498 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:57.218152 1498 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 32.58, Last appended by leader: 58, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.218237 1498 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 32 FOLLOWER]: Advancing to term 33
I20260501 08:10:57.333065 1480 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 32 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:57.333173 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 32 FOLLOWER]: Advancing to term 33
I20260501 08:10:57.333827 1480 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.334035 1480 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 33 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:57.334278 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 33 candidate_status { last_received { term: 32 index: 58 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:57.334335 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 33 candidate_status { last_received { term: 32 index: 58 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:57.334362 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 33: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:57.335199 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 33.
I20260501 08:10:57.335418 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:57.335542 1480 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 33 FOLLOWER]: Leader election won for term 33
I20260501 08:10:57.335608 1480 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 33 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:57.335642 1480 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 32.58, Last appended by leader: 58, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.336279 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 32 to 33, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 33 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:57.364024 1502 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:57.364167 1502 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 33 FOLLOWER]: Advancing to term 34
I20260501 08:10:57.364813 1502 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.365082 1502 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 34 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:57.365296 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 34 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:57.365384 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 33 FOLLOWER]: Advancing to term 34
I20260501 08:10:57.365406 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 34 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:57.366050 1384 raft_consensus.cc:2410] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a for term 34 because replica has last-logged OpId of term: 32 index: 58, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 57.
I20260501 08:10:57.366273 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:57.366379 1502 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260501 08:10:57.412012 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 34 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 33. Current term is 34. Ops: []
I20260501 08:10:57.412354 1480 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20260501 08:10:57.412528 1500 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 33 LEADER]: Stepping down as leader of term 33
I20260501 08:10:57.412580 1500 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 33 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:57.412679 1500 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 33.59, Last appended by leader: 59, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.412762 1500 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 33 FOLLOWER]: Advancing to term 34
I20260501 08:10:57.443745 1498 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:57.443853 1498 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 34 FOLLOWER]: Advancing to term 35
I20260501 08:10:57.444545 1498 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.444736 1498 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 35 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:57.445000 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 35 candidate_status { last_received { term: 32 index: 58 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:57.445000 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 35 candidate_status { last_received { term: 32 index: 58 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:57.445123 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 34 FOLLOWER]: Advancing to term 35
I20260501 08:10:57.445145 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 34 FOLLOWER]: Advancing to term 35
I20260501 08:10:57.445806 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 35.
I20260501 08:10:57.445819 1233 raft_consensus.cc:2410] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 for term 35 because replica has last-logged OpId of term: 33 index: 59, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 58.
I20260501 08:10:57.446091 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:57.446296 1498 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 35 FOLLOWER]: Leader election won for term 35
I20260501 08:10:57.446379 1498 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 35 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:57.446424 1498 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 32.58, Last appended by leader: 58, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.446998 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 33 to 35, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 35 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:57.532617 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 35 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 31 index: 57. Preceding OpId from leader: term: 35 index: 59. (index mismatch)
I20260501 08:10:57.532997 1498 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20260501 08:10:57.534245 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 35 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 33 index: 59. Preceding OpId from leader: term: 35 index: 59. (term mismatch)
I20260501 08:10:57.534304 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 58
I20260501 08:10:57.534324 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 33.59
I20260501 08:10:57.534341 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:57.534583 1503 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20260501 08:10:57.635288 1502 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 35 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:57.635406 1502 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 35 FOLLOWER]: Advancing to term 36
I20260501 08:10:57.636157 1502 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.636373 1502 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 36 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:57.636566 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 36 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:57.636566 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 36 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:57.637046 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:57.637184 1502 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260501 08:10:57.650385 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 36 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 35. Current term is 36. Ops: []
I20260501 08:10:57.650751 1495 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 35.59, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20260501 08:10:57.650986 1503 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 35 LEADER]: Stepping down as leader of term 35
I20260501 08:10:57.651051 1503 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:57.651110 1503 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 35.59, Last appended by leader: 59, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.651223 1503 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 35 FOLLOWER]: Advancing to term 36
I20260501 08:10:57.734814 1500 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 35 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:57.734973 1500 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 35 FOLLOWER]: Advancing to term 36
I20260501 08:10:57.735631 1500 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.735845 1500 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 36 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:57.736202 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 36 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:57.736301 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 36 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:57.736303 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 36: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:57.737104 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 36.
I20260501 08:10:57.737373 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:57.737475 1500 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 36 FOLLOWER]: Leader election won for term 36
I20260501 08:10:57.737562 1500 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 36 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:57.737627 1500 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 35.59, Last appended by leader: 59, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.738111 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 35 to 36, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 36 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:57.760694 1502 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:57.760802 1502 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 36 FOLLOWER]: Advancing to term 37
I20260501 08:10:57.761595 1502 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.761793 1502 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 37 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:57.762142 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 37 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:57.762138 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 37 candidate_status { last_received { term: 35 index: 59 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:57.762255 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 36 FOLLOWER]: Advancing to term 37
I20260501 08:10:57.762833 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 37.
I20260501 08:10:57.763062 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:57.763204 1502 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 37 FOLLOWER]: Leader election won for term 37
I20260501 08:10:57.763325 1502 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 37 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:57.763403 1502 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 35.59, Last appended by leader: 59, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.763919 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 36 to 37, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 37 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:57.856510 1233 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 36 LEADER]: Stepping down as leader of term 36
I20260501 08:10:57.856598 1233 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 36 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:57.856658 1233 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 36.60, Last appended by leader: 60, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:57.856805 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 36 FOLLOWER]: Advancing to term 37
I20260501 08:10:57.857617 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 37 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 36 index: 60. Preceding OpId from leader: term: 37 index: 60. (term mismatch)
I20260501 08:10:57.857663 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 59
I20260501 08:10:57.857676 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 36.60
I20260501 08:10:57.857690 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:57.858062 1502 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20260501 08:10:57.859396 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 37 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 35 index: 59. Preceding OpId from leader: term: 37 index: 60. (index mismatch)
I20260501 08:10:57.859591 1502 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20260501 08:10:58.044991 1503 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:10:58.045120 1503 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 37 FOLLOWER]: Advancing to term 38
I20260501 08:10:58.045928 1503 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.046177 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 38 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 37. Current term is 38. Ops: []
I20260501 08:10:58.046268 1503 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 38 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:58.046342 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 38 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:58.046355 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 38 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:58.046548 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 37.60, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20260501 08:10:58.046696 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 37 LEADER]: Stepping down as leader of term 37
I20260501 08:10:58.046586 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:58.046746 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:58.046798 1503 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260501 08:10:58.046819 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 37.60, Last appended by leader: 60, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.046943 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 37 FOLLOWER]: Advancing to term 38
I20260501 08:10:58.074688 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:58.082058 1500 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 37 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:10:58.082175 1500 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 37 FOLLOWER]: Advancing to term 38
I20260501 08:10:58.082978 1500 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.083168 1500 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 38 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:58.083395 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 38 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:58.083504 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 38 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:58.083657 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 38: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:10:58.084220 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 38.
I20260501 08:10:58.084445 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:58.084535 1500 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 38 FOLLOWER]: Leader election won for term 38
I20260501 08:10:58.084599 1500 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 38 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:58.084651 1500 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 37.60, Last appended by leader: 60, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.085281 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 37 to 38, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 38 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:58.161422 1503 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:58.161581 1503 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 38 FOLLOWER]: Advancing to term 39
I20260501 08:10:58.162300 1503 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.162513 1503 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 39 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:58.162730 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 39 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:58.162811 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 38 FOLLOWER]: Advancing to term 39
I20260501 08:10:58.162767 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 39 candidate_status { last_received { term: 37 index: 60 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:58.163465 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 39.
I20260501 08:10:58.163733 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:58.163853 1503 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 39 FOLLOWER]: Leader election won for term 39
I20260501 08:10:58.164047 1503 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 39 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:58.164155 1503 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 37.60, Last appended by leader: 60, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.164654 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 38 to 39, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 39 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:58.166520 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 39 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 38. Current term is 39. Ops: []
I20260501 08:10:58.166749 1500 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20260501 08:10:58.166851 1500 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 39.61 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 60
I20260501 08:10:58.166882 1480 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 38 LEADER]: Stepping down as leader of term 38
I20260501 08:10:58.166929 1480 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 38 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:58.166990 1480 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 38.61, Last appended by leader: 61, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.167063 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 38 FOLLOWER]: Advancing to term 39
I20260501 08:10:58.254195 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 39 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 37 index: 60. Preceding OpId from leader: term: 39 index: 61. (index mismatch)
I20260501 08:10:58.254431 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 39 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 38 index: 61. Preceding OpId from leader: term: 39 index: 61. (term mismatch)
I20260501 08:10:58.254501 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 60
I20260501 08:10:58.254554 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 38.61
I20260501 08:10:58.254489 1503 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20260501 08:10:58.254587 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:58.254936 1503 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20260501 08:10:58.435472 1480 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 39 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:58.435590 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 39 FOLLOWER]: Advancing to term 40
I20260501 08:10:58.436340 1480 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.436533 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 40 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 39. Current term is 40. Ops: []
I20260501 08:10:58.436556 1480 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 40 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:58.436671 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 40 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:58.436709 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 40 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:58.436805 1506 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 39.61, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20260501 08:10:58.436885 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:58.436993 1506 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 39 LEADER]: Stepping down as leader of term 39
I20260501 08:10:58.437006 1480 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260501 08:10:58.437048 1506 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:58.437122 1506 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 39.61, Last appended by leader: 61, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.437247 1506 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 39 FOLLOWER]: Advancing to term 40
I20260501 08:10:58.530215 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 39 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:58.530366 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 39 FOLLOWER]: Advancing to term 40
I20260501 08:10:58.531123 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.531358 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 40 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:58.531587 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 40 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:58.531659 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 40 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:58.531796 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 40: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:10:58.532356 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 40.
I20260501 08:10:58.532572 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:58.532660 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 40 FOLLOWER]: Leader election won for term 40
I20260501 08:10:58.532737 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 40 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:58.532790 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 39.61, Last appended by leader: 61, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.533407 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 39 to 40, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 40 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:58.572479 1480 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:58.572584 1480 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 40 FOLLOWER]: Advancing to term 41
I20260501 08:10:58.573246 1480 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.573441 1480 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 41 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:58.573640 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 41 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:58.573673 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 41 candidate_status { last_received { term: 39 index: 61 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:58.573804 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 40 FOLLOWER]: Advancing to term 41
I20260501 08:10:58.574347 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 41.
I20260501 08:10:58.574544 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:58.574749 1480 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 41 FOLLOWER]: Leader election won for term 41
I20260501 08:10:58.574833 1480 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 41 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:58.574888 1480 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 39.61, Last appended by leader: 61, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.575393 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 40 to 41, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 41 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:58.634967 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 41 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 40. Current term is 41. Ops: []
I20260501 08:10:58.635356 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20260501 08:10:58.635493 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 40 LEADER]: Stepping down as leader of term 40
I20260501 08:10:58.635527 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:58.635596 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 40.62, Last appended by leader: 62, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.635692 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 40 FOLLOWER]: Advancing to term 41
I20260501 08:10:58.657255 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 41 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 40 index: 62. Preceding OpId from leader: term: 41 index: 62. (term mismatch)
I20260501 08:10:58.657338 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 61
I20260501 08:10:58.657353 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 40.62
I20260501 08:10:58.657369 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:58.657661 1480 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20260501 08:10:58.659081 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 41 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 39 index: 61. Preceding OpId from leader: term: 41 index: 62. (index mismatch)
I20260501 08:10:58.659504 1480 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20260501 08:10:58.943574 1506 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 41 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:58.943795 1506 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 41 FOLLOWER]: Advancing to term 42
I20260501 08:10:58.944562 1506 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.944775 1506 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 42 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:58.944973 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:58.945009 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:58.945225 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:58.945329 1506 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260501 08:10:58.964507 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 42 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 41. Current term is 42. Ops: []
I20260501 08:10:58.964918 1480 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 41.62, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20260501 08:10:58.965101 1500 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 41 LEADER]: Stepping down as leader of term 41
I20260501 08:10:58.965142 1500 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 41 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:58.965245 1500 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 41.62, Last appended by leader: 62, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:58.965376 1500 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 41 FOLLOWER]: Advancing to term 42
I20260501 08:10:59.040838 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 41 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:59.040948 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 41 FOLLOWER]: Advancing to term 42
I20260501 08:10:59.041671 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.041863 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 42 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:59.042109 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:59.042122 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 42 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:59.042212 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 42: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:10:59.042743 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 42.
I20260501 08:10:59.042943 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:59.043056 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 42 FOLLOWER]: Leader election won for term 42
I20260501 08:10:59.043134 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 42 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:59.043196 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 41.62, Last appended by leader: 62, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.043694 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 41 to 42, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 42 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:59.062842 1506 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:59.062985 1506 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 42 FOLLOWER]: Advancing to term 43
I20260501 08:10:59.063767 1506 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.064091 1506 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 43 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:59.064381 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 43 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:59.064383 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 43 candidate_status { last_received { term: 41 index: 62 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:59.064520 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 42 FOLLOWER]: Advancing to term 43
I20260501 08:10:59.065124 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 43.
I20260501 08:10:59.065384 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:59.065495 1506 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 43 FOLLOWER]: Leader election won for term 43
I20260501 08:10:59.065605 1506 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 43 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:59.065697 1506 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 41.62, Last appended by leader: 62, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.066201 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 42 to 43, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 43 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:59.074992 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:10:59.147420 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 43 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 42. Current term is 43. Ops: []
I20260501 08:10:59.147739 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20260501 08:10:59.147890 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 42 LEADER]: Stepping down as leader of term 42
I20260501 08:10:59.147927 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:59.147995 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 42.63, Last appended by leader: 63, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.148088 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 42 FOLLOWER]: Advancing to term 43
I20260501 08:10:59.166819 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 43 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 41 index: 62. Preceding OpId from leader: term: 43 index: 63. (index mismatch)
I20260501 08:10:59.167192 1506 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20260501 08:10:59.168236 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 43 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 42 index: 63. Preceding OpId from leader: term: 43 index: 63. (term mismatch)
I20260501 08:10:59.168292 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 62
I20260501 08:10:59.168313 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 42.63
I20260501 08:10:59.168330 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:10:59.168507 1506 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20260501 08:10:59.268724 1500 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 43 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:10:59.268890 1500 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 43 FOLLOWER]: Advancing to term 44
I20260501 08:10:59.269912 1500 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.270198 1500 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 44 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:59.270460 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 44 candidate_status { last_received { term: 43 index: 63 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:59.270566 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 43 FOLLOWER]: Advancing to term 44
I20260501 08:10:59.270460 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 44 candidate_status { last_received { term: 43 index: 63 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:59.271332 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 44.
I20260501 08:10:59.271574 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:59.271683 1500 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 44 FOLLOWER]: Leader election won for term 44
I20260501 08:10:59.271770 1500 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 44 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:59.271824 1500 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 43.63, Last appended by leader: 63, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.272408 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 43 to 44, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 44 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:10:59.291671 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 44 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 43. Current term is 44. Ops: []
I20260501 08:10:59.291986 1506 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 43.63, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20260501 08:10:59.292331 1506 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 44.64 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 63
I20260501 08:10:59.292428 1508 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 43 LEADER]: Stepping down as leader of term 43
I20260501 08:10:59.292500 1508 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:59.292585 1508 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 43.63, Last appended by leader: 63, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.292695 1508 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 43 FOLLOWER]: Advancing to term 44
I20260501 08:10:59.368340 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 44 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 43 index: 63. Preceding OpId from leader: term: 44 index: 64. (index mismatch)
I20260501 08:10:59.368656 1500 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20260501 08:10:59.369861 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 44 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 43 index: 63. Preceding OpId from leader: term: 44 index: 64. (index mismatch)
I20260501 08:10:59.370087 1500 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20260501 08:10:59.470996 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 44 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:59.471127 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 44 FOLLOWER]: Advancing to term 45
I20260501 08:10:59.471844 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.472195 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 45 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:59.472465 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 45 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:59.472465 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 45 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:59.472852 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:10:59.472965 1504 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260501 08:10:59.495249 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 45 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 44. Current term is 45. Ops: []
I20260501 08:10:59.495657 1480 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 44.64, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20260501 08:10:59.495823 1500 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 44 LEADER]: Stepping down as leader of term 44
I20260501 08:10:59.495877 1500 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 44 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:10:59.495987 1500 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 44.64, Last appended by leader: 64, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.496059 1500 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 44 FOLLOWER]: Advancing to term 45
I20260501 08:10:59.562264 1508 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 44 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:10:59.562371 1508 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 44 FOLLOWER]: Advancing to term 45
I20260501 08:10:59.563140 1508 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.563402 1508 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 45 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:59.563699 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 45 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:59.563717 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 45 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:59.563836 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 45: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:10:59.564502 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 45.
I20260501 08:10:59.564703 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:59.564827 1508 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 45 FOLLOWER]: Leader election won for term 45
I20260501 08:10:59.564877 1508 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 45 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:59.564934 1508 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 44.64, Last appended by leader: 64, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.565521 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 44 to 45, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 45 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:59.620366 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:59.620468 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 45 FOLLOWER]: Advancing to term 46
I20260501 08:10:59.621142 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.621344 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 46 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:59.621547 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 46 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:59.621577 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 46 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:59.621711 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 45 FOLLOWER]: Advancing to term 46
I20260501 08:10:59.622354 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 46.
I20260501 08:10:59.622663 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:59.622828 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 46 FOLLOWER]: Leader election won for term 46
I20260501 08:10:59.622920 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 46 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:59.622974 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 44.64, Last appended by leader: 64, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.623544 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 45 to 46, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 46 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:59.651261 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 46 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 45. Current term is 46. Ops: []
I20260501 08:10:59.651660 1508 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20260501 08:10:59.651823 1508 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 46.65 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 64
I20260501 08:10:59.651866 1506 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 45 LEADER]: Stepping down as leader of term 45
I20260501 08:10:59.651957 1506 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:59.652012 1506 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 45.65, Last appended by leader: 65, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.652174 1506 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 45 FOLLOWER]: Advancing to term 46
I20260501 08:10:59.725847 1500 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:59.725976 1500 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 46 FOLLOWER]: Advancing to term 47
I20260501 08:10:59.726678 1500 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.727008 1500 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 47 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:59.727119 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 47 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:59.727154 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 47 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:59.727221 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 46 FOLLOWER]: Advancing to term 47
I20260501 08:10:59.727828 1384 raft_consensus.cc:2410] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da for term 47 because replica has last-logged OpId of term: 45 index: 65, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 64.
I20260501 08:10:59.728165 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:59.728276 1500 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260501 08:10:59.735472 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 47 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 46. Current term is 47. Ops: []
I20260501 08:10:59.735839 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20260501 08:10:59.735975 1504 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 45.65 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 64
I20260501 08:10:59.736012 1505 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 46 LEADER]: Stepping down as leader of term 46
I20260501 08:10:59.736058 1505 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:59.736132 1505 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 46.65, Last appended by leader: 65, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.736261 1505 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 46 FOLLOWER]: Advancing to term 47
I20260501 08:10:59.860749 1500 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:59.860860 1500 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 47 FOLLOWER]: Advancing to term 48
I20260501 08:10:59.861589 1500 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.861879 1500 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 48 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:10:59.862011 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 48 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:59.862116 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 47 FOLLOWER]: Advancing to term 48
I20260501 08:10:59.862015 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 48 candidate_status { last_received { term: 44 index: 64 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:59.862190 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 47 FOLLOWER]: Advancing to term 48
I20260501 08:10:59.862862 1310 raft_consensus.cc:2410] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da for term 48 because replica has last-logged OpId of term: 46 index: 65, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 64.
I20260501 08:10:59.862883 1384 raft_consensus.cc:2410] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da for term 48 because replica has last-logged OpId of term: 45 index: 65, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 64.
I20260501 08:10:59.863149 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:10:59.863267 1500 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260501 08:10:59.893874 1506 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:59.893992 1506 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 48 FOLLOWER]: Advancing to term 49
I20260501 08:10:59.894701 1506 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.894894 1506 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 49 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:59.895133 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 49 candidate_status { last_received { term: 45 index: 65 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:59.895224 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 49 candidate_status { last_received { term: 45 index: 65 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:10:59.895357 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 48 FOLLOWER]: Advancing to term 49
I20260501 08:10:59.895241 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 48 FOLLOWER]: Advancing to term 49
I20260501 08:10:59.895960 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 49.
I20260501 08:10:59.895983 1310 raft_consensus.cc:2410] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 for term 49 because replica has last-logged OpId of term: 46 index: 65, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 65.
I20260501 08:10:59.896236 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters:
I20260501 08:10:59.896341 1506 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 49 FOLLOWER]: Leader election won for term 49
I20260501 08:10:59.896425 1506 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 49 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:59.896484 1506 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 45.65, Last appended by leader: 65, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.897073 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 46 to 49, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 49 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:59.962914 1505 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:10:59.963057 1505 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 49 FOLLOWER]: Advancing to term 50
I20260501 08:10:59.963737 1505 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.963924 1505 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 50 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:10:59.964193 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 50 candidate_status { last_received { term: 46 index: 65 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:10:59.964160 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 50 candidate_status { last_received { term: 46 index: 65 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:10:59.964335 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 49 FOLLOWER]: Advancing to term 50
I20260501 08:10:59.964993 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 50.
I20260501 08:10:59.965255 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:10:59.965426 1505 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 50 FOLLOWER]: Leader election won for term 50
I20260501 08:10:59.965520 1505 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 50 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:10:59.965570 1505 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 46.65, Last appended by leader: 65, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.966183 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 49 to 50, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 50 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:10:59.999195 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 50 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 49. Current term is 50. Ops: []
I20260501 08:10:59.999504 1506 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20260501 08:10:59.999622 1506 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 50.66 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 64
I20260501 08:10:59.999650 1506 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 49 LEADER]: Stepping down as leader of term 49
I20260501 08:10:59.999686 1506 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:10:59.999763 1506 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 49.66, Last appended by leader: 66, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:10:59.999842 1506 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 49 FOLLOWER]: Advancing to term 50
I20260501 08:11:00.055172 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 50 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 49 index: 66. Preceding OpId from leader: term: 50 index: 66. (term mismatch)
I20260501 08:11:00.055303 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 65
I20260501 08:11:00.055327 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 49.66
I20260501 08:11:00.055351 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 50 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:00.055756 1505 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20260501 08:11:00.055851 1505 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 45.65 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 64
I20260501 08:11:00.056257 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 64
I20260501 08:11:00.056332 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 45.65
I20260501 08:11:00.056358 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 50 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:00.057399 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 50 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 44 index: 64. Preceding OpId from leader: term: 50 index: 66. (index mismatch)
I20260501 08:11:00.057576 1509 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20260501 08:11:00.075281 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:00.269668 1506 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 50 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:00.269779 1506 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 50 FOLLOWER]: Advancing to term 51
I20260501 08:11:00.270542 1506 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.270732 1506 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 51 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:00.271010 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 51 candidate_status { last_received { term: 50 index: 66 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:00.271027 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 51 candidate_status { last_received { term: 50 index: 66 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:00.271140 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 50 FOLLOWER]: Advancing to term 51
I20260501 08:11:00.271775 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 51.
I20260501 08:11:00.271981 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:00.272078 1506 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 51 FOLLOWER]: Leader election won for term 51
I20260501 08:11:00.272203 1506 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 51 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:00.272261 1506 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 50.66, Last appended by leader: 66, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.272735 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 50 to 51, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 51 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:00.285595 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 51 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 50. Current term is 51. Ops: []
I20260501 08:11:00.285924 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 50.66, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20260501 08:11:00.286124 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 50 LEADER]: Stepping down as leader of term 50
I20260501 08:11:00.286139 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 51 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 50. Current term is 51. Ops: []
I20260501 08:11:00.286185 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:00.286250 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 50.66, Last appended by leader: 66, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.286334 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 50 FOLLOWER]: Advancing to term 51
I20260501 08:11:00.383410 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 51 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 50 index: 66. Preceding OpId from leader: term: 51 index: 67. (index mismatch)
I20260501 08:11:00.383790 1506 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20260501 08:11:00.385128 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 51 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 50 index: 66. Preceding OpId from leader: term: 51 index: 67. (index mismatch)
I20260501 08:11:00.385440 1512 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20260501 08:11:00.486634 1510 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 51 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:00.486757 1510 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 51 FOLLOWER]: Advancing to term 52
I20260501 08:11:00.487506 1510 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.487717 1510 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 52 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:00.487929 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 52 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:00.487953 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 52 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:00.488308 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:00.488412 1510 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20260501 08:11:00.491739 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 52 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 51. Current term is 52. Ops: []
I20260501 08:11:00.492193 1508 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 51.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20260501 08:11:00.492345 1512 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 51 LEADER]: Stepping down as leader of term 51
I20260501 08:11:00.492398 1512 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:00.492484 1512 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 51.67, Last appended by leader: 67, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.492602 1512 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 51 FOLLOWER]: Advancing to term 52
I20260501 08:11:00.564028 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 51 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:00.564163 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 51 FOLLOWER]: Advancing to term 52
I20260501 08:11:00.564956 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.565153 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 52 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:00.565393 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 52 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:00.565397 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 52 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:00.565501 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 52: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:00.566170 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 52.
I20260501 08:11:00.566370 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:00.566480 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 52 FOLLOWER]: Leader election won for term 52
I20260501 08:11:00.566535 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 52 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:00.566591 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 51.67, Last appended by leader: 67, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.567446 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 51 to 52, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 52 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:00.627118 1510 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:00.627239 1510 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 52 FOLLOWER]: Advancing to term 53
I20260501 08:11:00.627954 1510 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.628172 1510 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 53 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:00.628407 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 53 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:00.628408 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 53 candidate_status { last_received { term: 51 index: 67 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:00.628533 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 52 FOLLOWER]: Advancing to term 53
I20260501 08:11:00.629370 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 53.
I20260501 08:11:00.629787 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:00.630033 1510 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 53 FOLLOWER]: Leader election won for term 53
I20260501 08:11:00.630159 1510 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 53 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:00.630232 1510 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 51.67, Last appended by leader: 67, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.630847 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 52 to 53, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 53 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:00.647607 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 53 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 52. Current term is 53. Ops: []
I20260501 08:11:00.648030 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20260501 08:11:00.648227 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 52 LEADER]: Stepping down as leader of term 52
I20260501 08:11:00.648272 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:00.648344 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 52.68, Last appended by leader: 68, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.648424 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 52 FOLLOWER]: Advancing to term 53
I20260501 08:11:00.707435 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 53 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 51 index: 67. Preceding OpId from leader: term: 53 index: 68. (index mismatch)
I20260501 08:11:00.707756 1510 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20260501 08:11:00.709089 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 53 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 52 index: 68. Preceding OpId from leader: term: 53 index: 68. (term mismatch)
I20260501 08:11:00.709148 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 67
I20260501 08:11:00.709172 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 52.68
I20260501 08:11:00.709192 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 53 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:00.709388 1514 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20260501 08:11:00.910423 1512 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 53 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:00.910545 1512 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 53 FOLLOWER]: Advancing to term 54
I20260501 08:11:00.911319 1512 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.911518 1512 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 54 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:00.911737 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 54 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:00.911811 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 54 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:00.912236 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:00.912418 1512 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20260501 08:11:00.931973 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 54 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 53. Current term is 54. Ops: []
I20260501 08:11:00.932379 1514 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 53.68, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20260501 08:11:00.932546 1513 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 53 LEADER]: Stepping down as leader of term 53
I20260501 08:11:00.932590 1513 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 53 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:00.932664 1513 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 53.68, Last appended by leader: 68, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.932760 1513 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 53 FOLLOWER]: Advancing to term 54
I20260501 08:11:00.988214 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 53 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:00.988319 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 53 FOLLOWER]: Advancing to term 54
I20260501 08:11:00.988951 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.989176 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 54 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:00.989392 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 54 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:00.989423 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 54 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:00.989457 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 54: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:00.990199 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 54.
I20260501 08:11:00.990438 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:00.990535 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 54 FOLLOWER]: Leader election won for term 54
I20260501 08:11:00.990626 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 54 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:00.990695 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 53.68, Last appended by leader: 68, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:00.991266 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 53 to 54, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 54 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:01.024942 1512 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:01.025094 1512 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 54 FOLLOWER]: Advancing to term 55
I20260501 08:11:01.025804 1512 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.026073 1512 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 55 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:01.026343 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 55 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:01.026343 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 55 candidate_status { last_received { term: 53 index: 68 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:01.026481 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 54 FOLLOWER]: Advancing to term 55
I20260501 08:11:01.027143 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 55.
I20260501 08:11:01.027362 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:01.027477 1512 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 55 FOLLOWER]: Leader election won for term 55
I20260501 08:11:01.027587 1512 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 55 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:01.027633 1512 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 53.68, Last appended by leader: 68, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.028319 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 54 to 55, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 55 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:01.069300 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 55 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 54. Current term is 55. Ops: []
I20260501 08:11:01.069725 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20260501 08:11:01.069936 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 54 LEADER]: Stepping down as leader of term 54
I20260501 08:11:01.070019 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:01.070089 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 54.69, Last appended by leader: 69, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.070185 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 54 FOLLOWER]: Advancing to term 55
I20260501 08:11:01.075559 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:01.111909 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 55 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 53 index: 68. Preceding OpId from leader: term: 55 index: 69. (index mismatch)
I20260501 08:11:01.112331 1512 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20260501 08:11:01.113416 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 55 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 54 index: 69. Preceding OpId from leader: term: 55 index: 69. (term mismatch)
I20260501 08:11:01.113474 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 68
I20260501 08:11:01.113495 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 54.69
I20260501 08:11:01.113515 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 55 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:01.113705 1515 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20260501 08:11:01.296176 1513 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 55 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:01.296286 1513 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 55 FOLLOWER]: Advancing to term 56
I20260501 08:11:01.297050 1513 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.297255 1513 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 56 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:01.297447 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 56 candidate_status { last_received { term: 55 index: 69 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:01.297469 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 56 candidate_status { last_received { term: 55 index: 69 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:01.297761 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:01.297871 1513 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20260501 08:11:01.312368 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 56 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 55. Current term is 56. Ops: []
I20260501 08:11:01.312681 1516 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 55.69, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20260501 08:11:01.312810 1516 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 55 LEADER]: Stepping down as leader of term 55
I20260501 08:11:01.312829 1516 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 55 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:01.312911 1516 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 55.69, Last appended by leader: 69, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.313030 1516 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 55 FOLLOWER]: Advancing to term 56
I20260501 08:11:01.403221 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 55 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:01.403394 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 55 FOLLOWER]: Advancing to term 56
I20260501 08:11:01.404781 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.405082 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 56 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:01.405344 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 56 candidate_status { last_received { term: 55 index: 69 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:01.405364 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 56 candidate_status { last_received { term: 55 index: 69 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:01.405509 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 56: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:01.406222 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 56.
I20260501 08:11:01.406504 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:01.406615 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 56 FOLLOWER]: Leader election won for term 56
I20260501 08:11:01.406666 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 56 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:01.406713 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 55.69, Last appended by leader: 69, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.407459 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 55 to 56, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 56 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:01.428529 1513 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 56 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:01.428640 1513 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 56 FOLLOWER]: Advancing to term 57
I20260501 08:11:01.429352 1513 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.429566 1513 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 57 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:01.429921 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 57 candidate_status { last_received { term: 55 index: 69 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:01.429965 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 57 candidate_status { last_received { term: 55 index: 69 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:01.430047 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 56 FOLLOWER]: Advancing to term 57
I20260501 08:11:01.430719 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 57.
I20260501 08:11:01.430936 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:01.431092 1513 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 57 FOLLOWER]: Leader election won for term 57
I20260501 08:11:01.431164 1513 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 57 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:01.431243 1513 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 55.69, Last appended by leader: 69, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.431856 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 56 to 57, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 57 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:01.507717 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 57 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 56. Current term is 57. Ops: []
I20260501 08:11:01.508059 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20260501 08:11:01.508224 1504 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 57.70 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69
I20260501 08:11:01.508256 1505 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 56 LEADER]: Stepping down as leader of term 56
I20260501 08:11:01.508297 1505 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 56 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:01.508421 1505 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 56.70, Last appended by leader: 70, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.508504 1505 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 56 FOLLOWER]: Advancing to term 57
I20260501 08:11:01.516283 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 57 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 55 index: 69. Preceding OpId from leader: term: 57 index: 70. (index mismatch)
I20260501 08:11:01.516464 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 57 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 56 index: 70. Preceding OpId from leader: term: 57 index: 70. (term mismatch)
I20260501 08:11:01.516533 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 69
I20260501 08:11:01.516556 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 56.70
I20260501 08:11:01.516592 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 57 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:01.516577 1513 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20260501 08:11:01.516867 1513 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20260501 08:11:01.713147 1516 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 57 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:01.713244 1516 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 57 FOLLOWER]: Advancing to term 58
I20260501 08:11:01.714361 1516 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.714640 1516 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 58 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:01.714761 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 58 candidate_status { last_received { term: 57 index: 70 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:01.714784 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 58 candidate_status { last_received { term: 57 index: 70 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:01.715055 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 58 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:01.715154 1516 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 58 FOLLOWER]: Leader election lost for term 58. Reason: could not achieve majority
I20260501 08:11:01.716037 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 58 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 57. Current term is 58. Ops: []
I20260501 08:11:01.716315 1514 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 57.70, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20260501 08:11:01.716449 1513 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 57 LEADER]: Stepping down as leader of term 57
I20260501 08:11:01.716491 1513 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 57 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:01.716568 1513 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 57.70, Last appended by leader: 70, Current term: 57, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.716688 1513 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 57 FOLLOWER]: Advancing to term 58
I20260501 08:11:01.813549 1505 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 57 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:01.813665 1505 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 57 FOLLOWER]: Advancing to term 58
I20260501 08:11:01.814505 1505 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.814767 1505 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 58 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:01.814922 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 58 candidate_status { last_received { term: 57 index: 70 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:01.814997 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 58: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:01.815251 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 58 candidate_status { last_received { term: 57 index: 70 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:01.815897 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 58.
I20260501 08:11:01.816341 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:01.816464 1505 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 58 FOLLOWER]: Leader election won for term 58
I20260501 08:11:01.816543 1505 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 58 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:01.816610 1505 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 57.70, Last appended by leader: 70, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.817411 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 57 to 58, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 58 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:01.844717 1516 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 58 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:01.844825 1516 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 58 FOLLOWER]: Advancing to term 59
I20260501 08:11:01.845578 1516 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.845886 1516 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 59 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:01.846024 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 59 candidate_status { last_received { term: 57 index: 70 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:01.846103 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 59 candidate_status { last_received { term: 57 index: 70 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:01.846208 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 58 FOLLOWER]: Advancing to term 59
I20260501 08:11:01.846836 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 59.
I20260501 08:11:01.847123 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:01.847219 1516 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 59 FOLLOWER]: Leader election won for term 59
I20260501 08:11:01.847329 1516 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 59 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:01.847401 1516 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 57.70, Last appended by leader: 70, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.848095 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 58 to 59, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 59 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:01.910979 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 59 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 58. Current term is 59. Ops: []
I20260501 08:11:01.911293 1505 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20260501 08:11:01.911438 1505 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 59.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 70
I20260501 08:11:01.911473 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 58 LEADER]: Stepping down as leader of term 58
I20260501 08:11:01.911512 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 58 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:01.911587 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 58.71, Last appended by leader: 71, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:01.911667 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 58 FOLLOWER]: Advancing to term 59
I20260501 08:11:01.938414 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 59 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 58 index: 71. Preceding OpId from leader: term: 59 index: 71. (term mismatch)
I20260501 08:11:01.938517 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 70
I20260501 08:11:01.938537 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 58.71
I20260501 08:11:01.938561 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 59 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:01.938879 1516 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20260501 08:11:01.940361 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 59 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 57 index: 70. Preceding OpId from leader: term: 59 index: 71. (index mismatch)
I20260501 08:11:01.940613 1516 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20260501 08:11:02.041740 1513 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 59 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:02.041848 1513 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 59 FOLLOWER]: Advancing to term 60
I20260501 08:11:02.042675 1513 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.042974 1513 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 60 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:02.043076 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 60 candidate_status { last_received { term: 59 index: 71 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:02.043097 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 60 candidate_status { last_received { term: 59 index: 71 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:02.043412 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:02.043565 1513 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I20260501 08:11:02.062978 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 60 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 59. Current term is 60. Ops: []
I20260501 08:11:02.063273 1516 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 59.71, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20260501 08:11:02.063422 1518 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 59 LEADER]: Stepping down as leader of term 59
I20260501 08:11:02.063479 1518 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 59 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:02.063550 1518 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 59.71, Last appended by leader: 71, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.063669 1518 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 59 FOLLOWER]: Advancing to term 60
I20260501 08:11:02.075848 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:02.137286 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 59 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:02.137387 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 59 FOLLOWER]: Advancing to term 60
I20260501 08:11:02.138151 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.138366 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 60 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:02.138705 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 60 candidate_status { last_received { term: 59 index: 71 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:02.138705 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 60 candidate_status { last_received { term: 59 index: 71 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:02.138836 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 60: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:02.139546 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 60 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 60.
I20260501 08:11:02.139771 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 60 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:02.139912 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 60 FOLLOWER]: Leader election won for term 60
I20260501 08:11:02.139997 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 60 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:02.140087 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 59.71, Last appended by leader: 71, Current term: 60, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.140793 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 59 to 60, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 60 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:02.173659 1513 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 60 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:02.173789 1513 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 60 FOLLOWER]: Advancing to term 61
I20260501 08:11:02.174544 1513 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.174819 1513 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 61 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:02.175015 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 61 candidate_status { last_received { term: 59 index: 71 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:02.175047 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 61 candidate_status { last_received { term: 59 index: 71 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:02.175201 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 60 FOLLOWER]: Advancing to term 61
I20260501 08:11:02.175890 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 61.
I20260501 08:11:02.176412 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 61 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:02.176558 1513 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 61 FOLLOWER]: Leader election won for term 61
I20260501 08:11:02.176640 1513 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 61 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:02.176697 1513 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 59.71, Last appended by leader: 71, Current term: 61, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.177403 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 60 to 61, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 61 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:02.228531 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 61 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 60. Current term is 61. Ops: []
I20260501 08:11:02.228933 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20260501 08:11:02.229079 1504 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 61.72 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 71
I20260501 08:11:02.229151 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 60 LEADER]: Stepping down as leader of term 60
I20260501 08:11:02.229183 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 60 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:02.229252 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 60.72, Last appended by leader: 72, Current term: 60, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.229352 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 60 FOLLOWER]: Advancing to term 61
I20260501 08:11:02.282944 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 61 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 59 index: 71. Preceding OpId from leader: term: 61 index: 72. (index mismatch)
I20260501 08:11:02.283493 1513 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20260501 08:11:02.284783 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 61 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 60 index: 72. Preceding OpId from leader: term: 61 index: 72. (term mismatch)
I20260501 08:11:02.284840 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 71
I20260501 08:11:02.284862 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 60.72
I20260501 08:11:02.284891 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 61 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:02.285041 1513 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20260501 08:11:02.385641 1518 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 61 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:02.385762 1518 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 61 FOLLOWER]: Advancing to term 62
I20260501 08:11:02.386499 1518 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.386703 1518 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 62 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:02.386930 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 62 candidate_status { last_received { term: 61 index: 72 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:02.387013 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 62 candidate_status { last_received { term: 61 index: 72 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:02.387307 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 62 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:02.387419 1518 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 62 FOLLOWER]: Leader election lost for term 62. Reason: could not achieve majority
I20260501 08:11:02.398972 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 62 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 61. Current term is 62. Ops: []
I20260501 08:11:02.399300 1514 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 61.72, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20260501 08:11:02.399456 1514 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 61 LEADER]: Stepping down as leader of term 61
I20260501 08:11:02.399487 1514 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 61 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:02.399566 1514 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 61.72, Last appended by leader: 72, Current term: 61, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.399657 1514 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 61 FOLLOWER]: Advancing to term 62
I20260501 08:11:02.472525 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 61 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:02.472640 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 61 FOLLOWER]: Advancing to term 62
I20260501 08:11:02.473392 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.473551 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 62 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:02.473896 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 62 candidate_status { last_received { term: 61 index: 72 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:02.473951 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 62 candidate_status { last_received { term: 61 index: 72 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:02.473971 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 62 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 62: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:02.474721 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 62 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 62.
I20260501 08:11:02.475049 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 62 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:02.475184 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 62 FOLLOWER]: Leader election won for term 62
I20260501 08:11:02.475309 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 62 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:02.475383 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 61.72, Last appended by leader: 72, Current term: 62, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.475857 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 61 to 62, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 62 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:02.501820 1518 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 62 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:02.501977 1518 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 62 FOLLOWER]: Advancing to term 63
I20260501 08:11:02.502942 1518 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.503337 1518 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 63 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:02.503662 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 63 candidate_status { last_received { term: 61 index: 72 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:02.503885 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 62 FOLLOWER]: Advancing to term 63
I20260501 08:11:02.503662 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 63 candidate_status { last_received { term: 61 index: 72 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:02.504701 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 63.
I20260501 08:11:02.504910 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 63 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:02.505031 1518 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 63 FOLLOWER]: Leader election won for term 63
I20260501 08:11:02.505105 1518 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 63 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:02.505276 1518 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 61.72, Last appended by leader: 72, Current term: 63, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.505769 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 62 to 63, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 63 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:02.572512 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 63 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 62. Current term is 63. Ops: []
I20260501 08:11:02.572921 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20260501 08:11:02.573030 1504 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 63.73 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 72
I20260501 08:11:02.573057 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 62 LEADER]: Stepping down as leader of term 62
I20260501 08:11:02.573313 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 62 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:02.573390 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 62.73, Last appended by leader: 73, Current term: 62, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.573482 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 62 FOLLOWER]: Advancing to term 63
I20260501 08:11:02.608691 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 63 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 61 index: 72. Preceding OpId from leader: term: 63 index: 73. (index mismatch)
I20260501 08:11:02.609213 1518 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20260501 08:11:02.609424 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 63 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 62 index: 73. Preceding OpId from leader: term: 63 index: 73. (term mismatch)
I20260501 08:11:02.609488 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 72
I20260501 08:11:02.609509 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 62.73
I20260501 08:11:02.609539 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 63 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:02.609741 1518 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20260501 08:11:02.710973 1514 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 63 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:02.711122 1514 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 63 FOLLOWER]: Advancing to term 64
I20260501 08:11:02.711833 1514 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.712080 1514 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 64 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:02.712369 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 64 candidate_status { last_received { term: 63 index: 73 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:02.712378 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 64 candidate_status { last_received { term: 63 index: 73 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:02.712543 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 63 FOLLOWER]: Advancing to term 64
I20260501 08:11:02.713271 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 64 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 64.
I20260501 08:11:02.713522 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 64 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:02.713694 1514 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 64 FOLLOWER]: Leader election won for term 64
I20260501 08:11:02.713757 1514 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 64 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:02.713822 1514 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 63.73, Last appended by leader: 73, Current term: 64, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.714159 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 64 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 63. Current term is 64. Ops: []
I20260501 08:11:02.714224 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 64 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 63. Current term is 64. Ops: []
I20260501 08:11:02.714409 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 63 to 64, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 64 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:02.714434 1520 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 63.73, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20260501 08:11:02.714567 1520 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 64.74 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 73
I20260501 08:11:02.714612 1520 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 63 LEADER]: Stepping down as leader of term 63
I20260501 08:11:02.714634 1520 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 63 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:02.714692 1520 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 63.73, Last appended by leader: 73, Current term: 63, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.714757 1520 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 63 FOLLOWER]: Advancing to term 64
W20260501 08:11:02.715507 1522 consensus_queue.cc:1175] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4b9150123ff048d08f1d65634bf2425a" responder_term: 64 status { last_received { term: 63 index: 73 } last_committed_idx: 73 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 63. Current term is 64. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260501 08:11:02.823043 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 64 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:02.823170 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 64 FOLLOWER]: Advancing to term 65
I20260501 08:11:02.824033 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.824318 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 65 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:02.824533 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 65 candidate_status { last_received { term: 63 index: 73 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:02.824534 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 65 candidate_status { last_received { term: 63 index: 73 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:02.824632 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 64 FOLLOWER]: Advancing to term 65
I20260501 08:11:02.825249 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 65 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 65.
I20260501 08:11:02.825456 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 65 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:02.825558 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 65 FOLLOWER]: Leader election won for term 65
I20260501 08:11:02.825639 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 65 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:02.825693 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 63.73, Last appended by leader: 73, Current term: 65, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.826249 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 64 to 65, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 65 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:02.831709 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 65 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 64. Current term is 65. Ops: []
I20260501 08:11:02.832083 1514 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20260501 08:11:02.832248 1514 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 65.74 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 73
I20260501 08:11:02.832289 1519 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 64 LEADER]: Stepping down as leader of term 64
I20260501 08:11:02.832333 1519 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 64 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:02.832412 1519 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 64.74, Last appended by leader: 74, Current term: 64, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:02.832494 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 64 FOLLOWER]: Advancing to term 65
I20260501 08:11:02.906874 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 65 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 63 index: 73. Preceding OpId from leader: term: 65 index: 74. (index mismatch)
I20260501 08:11:02.907226 1504 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20260501 08:11:02.908650 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 65 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 64 index: 74. Preceding OpId from leader: term: 65 index: 74. (term mismatch)
I20260501 08:11:02.908730 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 73
I20260501 08:11:02.908751 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 64.74
I20260501 08:11:02.908771 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 65 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:02.908927 1523 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20260501 08:11:03.009267 1522 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 65 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:03.009368 1522 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 65 FOLLOWER]: Advancing to term 66
I20260501 08:11:03.010145 1522 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.010347 1522 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 66 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:03.010615 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 66 candidate_status { last_received { term: 65 index: 74 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:03.010650 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 66 candidate_status { last_received { term: 65 index: 74 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:03.011356 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:03.011500 1522 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20260501 08:11:03.029536 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 66 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 65. Current term is 66. Ops: []
I20260501 08:11:03.029942 1523 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 65.74, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20260501 08:11:03.030073 1523 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 65 LEADER]: Stepping down as leader of term 65
I20260501 08:11:03.030094 1523 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 65 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:03.030210 1523 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 65.74, Last appended by leader: 74, Current term: 65, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.030301 1523 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 65 FOLLOWER]: Advancing to term 66
I20260501 08:11:03.076165 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:03.107362 1519 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 65 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:03.107476 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 65 FOLLOWER]: Advancing to term 66
I20260501 08:11:03.108261 1519 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.108443 1519 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 66 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:03.108703 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 66 candidate_status { last_received { term: 65 index: 74 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:03.108793 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 66: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:03.108708 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 66 candidate_status { last_received { term: 65 index: 74 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:03.109566 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 66 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 66.
I20260501 08:11:03.109823 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 66 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:03.109930 1519 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 66 FOLLOWER]: Leader election won for term 66
I20260501 08:11:03.110026 1519 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 66 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:03.110087 1519 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 65.74, Last appended by leader: 74, Current term: 66, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.110642 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 65 to 66, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 66 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:03.153102 1522 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 66 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:03.153206 1522 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 66 FOLLOWER]: Advancing to term 67
I20260501 08:11:03.153885 1522 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.154206 1522 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 67 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:03.154328 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 67 candidate_status { last_received { term: 65 index: 74 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:03.154327 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 67 candidate_status { last_received { term: 65 index: 74 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:03.154428 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 66 FOLLOWER]: Advancing to term 67
I20260501 08:11:03.155112 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 67 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 67.
I20260501 08:11:03.155359 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 67 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:03.155478 1522 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 67 FOLLOWER]: Leader election won for term 67
I20260501 08:11:03.155529 1522 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 67 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:03.155608 1522 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 65.74, Last appended by leader: 74, Current term: 67, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.156267 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 66 to 67, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 67 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:03.210343 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 67 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 66. Current term is 67. Ops: []
I20260501 08:11:03.210664 1519 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20260501 08:11:03.210795 1519 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 67.75 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 74
I20260501 08:11:03.210829 1514 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 66 LEADER]: Stepping down as leader of term 66
I20260501 08:11:03.210877 1514 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 66 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:03.210947 1514 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 66.75, Last appended by leader: 75, Current term: 66, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.211059 1514 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 66 FOLLOWER]: Advancing to term 67
I20260501 08:11:03.261755 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 67 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 66 index: 75. Preceding OpId from leader: term: 67 index: 75. (term mismatch)
I20260501 08:11:03.261873 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 74
I20260501 08:11:03.261895 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 66.75
I20260501 08:11:03.261917 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 67 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:03.262259 1522 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20260501 08:11:03.263283 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 67 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 65 index: 74. Preceding OpId from leader: term: 67 index: 75. (index mismatch)
I20260501 08:11:03.263510 1524 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20260501 08:11:03.441037 1514 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 67 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:03.441152 1514 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 67 FOLLOWER]: Advancing to term 68
I20260501 08:11:03.441849 1514 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.442060 1514 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 68 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:03.442289 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 68 candidate_status { last_received { term: 67 index: 75 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:03.442289 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 68 candidate_status { last_received { term: 67 index: 75 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:03.442574 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 68 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:03.442682 1514 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 68 FOLLOWER]: Leader election lost for term 68. Reason: could not achieve majority
I20260501 08:11:03.461820 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 68 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 67. Current term is 68. Ops: []
I20260501 08:11:03.462116 1520 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 67.75, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260501 08:11:03.462265 1524 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 67 LEADER]: Stepping down as leader of term 67
I20260501 08:11:03.462311 1524 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 67 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:03.462392 1524 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 67.75, Last appended by leader: 75, Current term: 67, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.462479 1524 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 67 FOLLOWER]: Advancing to term 68
I20260501 08:11:03.463719 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 67 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:03.463783 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 67 FOLLOWER]: Advancing to term 68
I20260501 08:11:03.464404 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.464560 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 68 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:03.464733 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 68 candidate_status { last_received { term: 67 index: 75 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:03.464742 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 68 candidate_status { last_received { term: 67 index: 75 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:03.464852 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 68 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 68: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:03.465425 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 68 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 68.
I20260501 08:11:03.465621 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 68 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:03.465718 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 68 FOLLOWER]: Leader election won for term 68
I20260501 08:11:03.465790 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 68 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:03.465837 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 67.75, Last appended by leader: 75, Current term: 68, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.466426 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 67 to 68, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 68 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:03.548034 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 68 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 67 index: 75. Preceding OpId from leader: term: 68 index: 76. (index mismatch)
I20260501 08:11:03.548408 1504 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260501 08:11:03.549701 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 68 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 67 index: 75. Preceding OpId from leader: term: 68 index: 76. (index mismatch)
I20260501 08:11:03.549919 1504 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20260501 08:11:03.651046 1519 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 68 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:03.651155 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 68 FOLLOWER]: Advancing to term 69
I20260501 08:11:03.651818 1519 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.652024 1519 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 69 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:03.652298 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 69 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:03.652298 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 69 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:03.652592 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 69 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:03.652771 1519 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 69 FOLLOWER]: Leader election lost for term 69. Reason: could not achieve majority
I20260501 08:11:03.664274 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 69 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 68. Current term is 69. Ops: []
I20260501 08:11:03.664582 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 68.76, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20260501 08:11:03.664700 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 68 LEADER]: Stepping down as leader of term 68
I20260501 08:11:03.664721 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 68 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:03.664781 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 68.76, Last appended by leader: 76, Current term: 68, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.664848 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 68 FOLLOWER]: Advancing to term 69
I20260501 08:11:03.726526 1524 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 68 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:03.726652 1524 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 68 FOLLOWER]: Advancing to term 69
I20260501 08:11:03.727491 1524 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.727702 1524 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 69 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:03.728170 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 69 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:03.728178 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 69 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:03.728335 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 69 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 69: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:03.728909 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 69 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 69.
I20260501 08:11:03.729146 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 69 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:03.729238 1524 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 69 FOLLOWER]: Leader election won for term 69
I20260501 08:11:03.729283 1524 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 69 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:03.729333 1524 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 68.76, Last appended by leader: 76, Current term: 69, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.729791 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 68 to 69, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 69 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:03.778354 1519 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 69 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:03.778455 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 69 FOLLOWER]: Advancing to term 70
I20260501 08:11:03.779119 1519 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.779378 1519 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 70 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:03.779582 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 70 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:03.779582 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 70 candidate_status { last_received { term: 68 index: 76 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:03.779700 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 69 FOLLOWER]: Advancing to term 70
I20260501 08:11:03.780356 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 70 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 70.
I20260501 08:11:03.780550 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 70 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:03.780675 1519 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 70 FOLLOWER]: Leader election won for term 70
I20260501 08:11:03.780732 1519 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 70 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:03.780793 1519 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 68.76, Last appended by leader: 76, Current term: 70, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.781337 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 69 to 70, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 70 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:03.809521 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 70 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 69. Current term is 70. Ops: []
I20260501 08:11:03.809909 1524 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20260501 08:11:03.810077 1524 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 70.77 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 76
I20260501 08:11:03.810137 1520 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 69 LEADER]: Stepping down as leader of term 69
I20260501 08:11:03.810201 1520 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 69 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:03.810293 1520 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 69.77, Last appended by leader: 77, Current term: 69, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.810389 1520 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 69 FOLLOWER]: Advancing to term 70
I20260501 08:11:03.861600 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 70 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 69 index: 77. Preceding OpId from leader: term: 70 index: 77. (term mismatch)
I20260501 08:11:03.861680 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 76
I20260501 08:11:03.861694 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 69.77
I20260501 08:11:03.861708 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 70 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:03.861996 1519 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20260501 08:11:03.863404 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 70 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 68 index: 76. Preceding OpId from leader: term: 70 index: 77. (index mismatch)
I20260501 08:11:03.863873 1525 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20260501 08:11:03.965603 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 70 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:03.965724 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 70 FOLLOWER]: Advancing to term 71
I20260501 08:11:03.966408 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.966802 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 71 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:03.967022 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 71 candidate_status { last_received { term: 70 index: 77 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:03.967098 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 71 candidate_status { last_received { term: 70 index: 77 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:03.967355 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 71 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:03.967456 1504 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 71 FOLLOWER]: Leader election lost for term 71. Reason: could not achieve majority
I20260501 08:11:03.970687 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 71 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 70. Current term is 71. Ops: []
I20260501 08:11:03.971215 1514 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 70.77, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20260501 08:11:03.971374 1519 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 70 LEADER]: Stepping down as leader of term 70
I20260501 08:11:03.971411 1519 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 70 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:03.971468 1519 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 70.77, Last appended by leader: 77, Current term: 70, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:03.971521 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 70 FOLLOWER]: Advancing to term 71
I20260501 08:11:04.044265 1520 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 70 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:04.044373 1520 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 70 FOLLOWER]: Advancing to term 71
I20260501 08:11:04.045020 1520 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.045215 1520 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 71 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:04.045436 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 71 candidate_status { last_received { term: 70 index: 77 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:04.045531 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 71 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 71: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:04.045488 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 71 candidate_status { last_received { term: 70 index: 77 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:04.046201 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 71 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 71.
I20260501 08:11:04.046415 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 71 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:04.046521 1520 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 71 FOLLOWER]: Leader election won for term 71
I20260501 08:11:04.046582 1520 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 71 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:04.046643 1520 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 70.77, Last appended by leader: 77, Current term: 71, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.047335 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 70 to 71, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 71 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:04.076491 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:04.079978 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 71 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:04.080077 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 71 FOLLOWER]: Advancing to term 72
I20260501 08:11:04.080741 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.080932 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 72 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:04.081195 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 72 candidate_status { last_received { term: 70 index: 77 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:04.081228 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 72 candidate_status { last_received { term: 70 index: 77 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:04.081348 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 71 FOLLOWER]: Advancing to term 72
I20260501 08:11:04.081916 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 72 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 72.
I20260501 08:11:04.082149 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 72 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:04.082310 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 72 FOLLOWER]: Leader election won for term 72
I20260501 08:11:04.082378 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 72 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:04.082463 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 70.77, Last appended by leader: 77, Current term: 72, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.083024 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 71 to 72, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 72 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:04.122648 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 72 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 71. Current term is 72. Ops: []
I20260501 08:11:04.123095 1520 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20260501 08:11:04.123256 1520 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 72.78 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 77
I20260501 08:11:04.123293 1524 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 71 LEADER]: Stepping down as leader of term 71
I20260501 08:11:04.123342 1524 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 71 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:04.123410 1524 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 71.78, Last appended by leader: 78, Current term: 71, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.123492 1524 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 71 FOLLOWER]: Advancing to term 72
I20260501 08:11:04.179540 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 72 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 70 index: 77. Preceding OpId from leader: term: 72 index: 78. (index mismatch)
I20260501 08:11:04.180007 1504 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20260501 08:11:04.181198 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 72 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 71 index: 78. Preceding OpId from leader: term: 72 index: 78. (term mismatch)
I20260501 08:11:04.181257 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 77
I20260501 08:11:04.181279 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 71.78
I20260501 08:11:04.181310 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 72 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:04.181644 1526 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20260501 08:11:04.481813 1524 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 72 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:04.481931 1524 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 72 FOLLOWER]: Advancing to term 73
I20260501 08:11:04.482686 1524 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.482808 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 73 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 72. Current term is 73. Ops: []
I20260501 08:11:04.482947 1524 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 73 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:04.483037 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 73 candidate_status { last_received { term: 72 index: 78 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:04.483026 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 73 candidate_status { last_received { term: 72 index: 78 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:04.483114 1526 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 72.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20260501 08:11:04.483235 1504 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 72 LEADER]: Stepping down as leader of term 72
I20260501 08:11:04.483255 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 73 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:04.483274 1504 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 72 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:04.483331 1504 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 72.78, Last appended by leader: 78, Current term: 72, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.483433 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 72 FOLLOWER]: Advancing to term 73
I20260501 08:11:04.483434 1524 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 73 FOLLOWER]: Leader election lost for term 73. Reason: could not achieve majority
I20260501 08:11:04.551656 1519 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 72 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:04.551770 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 72 FOLLOWER]: Advancing to term 73
I20260501 08:11:04.552636 1519 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.552831 1519 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 73 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:04.553113 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 73 candidate_status { last_received { term: 72 index: 78 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:04.553154 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 73 candidate_status { last_received { term: 72 index: 78 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:04.553274 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 73 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 73: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:04.553834 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 73.
I20260501 08:11:04.554085 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 73 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:04.554224 1519 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 73 FOLLOWER]: Leader election won for term 73
I20260501 08:11:04.554376 1519 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 73 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:04.554445 1519 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 72.78, Last appended by leader: 78, Current term: 73, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.555056 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 72 to 73, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 73 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:04.605811 1524 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 73 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:04.605950 1524 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 73 FOLLOWER]: Advancing to term 74
I20260501 08:11:04.606604 1524 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.606817 1524 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 74 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:04.607024 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 74 candidate_status { last_received { term: 72 index: 78 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:04.607097 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 74 candidate_status { last_received { term: 72 index: 78 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:04.607143 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 73 FOLLOWER]: Advancing to term 74
I20260501 08:11:04.607722 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 74 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 74.
I20260501 08:11:04.607935 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 74 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:04.608031 1524 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 74 FOLLOWER]: Leader election won for term 74
I20260501 08:11:04.608109 1524 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 74 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:04.608180 1524 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 72.78, Last appended by leader: 78, Current term: 74, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.608644 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 73 to 74, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 74 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:04.639461 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 74 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 73. Current term is 74. Ops: []
I20260501 08:11:04.639766 1519 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20260501 08:11:04.639904 1519 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 73 LEADER]: Stepping down as leader of term 73
I20260501 08:11:04.639938 1519 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 73 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:04.639999 1519 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 73.79, Last appended by leader: 79, Current term: 73, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.640089 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 73 FOLLOWER]: Advancing to term 74
I20260501 08:11:04.687372 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 74 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 73 index: 79. Preceding OpId from leader: term: 74 index: 79. (term mismatch)
I20260501 08:11:04.687453 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 78
I20260501 08:11:04.687467 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 73.79
I20260501 08:11:04.687481 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 74 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:04.687770 1524 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20260501 08:11:04.688978 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 74 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 72 index: 78. Preceding OpId from leader: term: 74 index: 79. (index mismatch)
I20260501 08:11:04.689211 1528 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20260501 08:11:04.866781 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 74 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:04.866894 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 74 FOLLOWER]: Advancing to term 75
I20260501 08:11:04.867614 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.867802 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 75 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:04.868152 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 75 candidate_status { last_received { term: 74 index: 79 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:04.868152 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 75 candidate_status { last_received { term: 74 index: 79 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:04.868412 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 75 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:04.868543 1504 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 75 FOLLOWER]: Leader election lost for term 75. Reason: could not achieve majority
I20260501 08:11:04.875517 1519 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 74 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:04.875617 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 74 FOLLOWER]: Advancing to term 75
I20260501 08:11:04.876447 1519 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.876612 1519 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 75 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:04.876823 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 75 candidate_status { last_received { term: 74 index: 79 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:04.876823 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 75 candidate_status { last_received { term: 74 index: 79 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:04.876955 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 75 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 75: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:04.877120 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 75 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:04.877225 1519 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 75 FOLLOWER]: Leader election lost for term 75. Reason: could not achieve majority
I20260501 08:11:04.880842 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 75 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 74. Current term is 75. Ops: []
I20260501 08:11:04.881134 1520 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 74.79, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20260501 08:11:04.881290 1520 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 74 LEADER]: Stepping down as leader of term 74
I20260501 08:11:04.881356 1520 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 74 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:04.881424 1520 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 74.79, Last appended by leader: 79, Current term: 74, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.881531 1520 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 74 FOLLOWER]: Advancing to term 75
I20260501 08:11:04.994774 1519 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 75 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:04.994889 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 75 FOLLOWER]: Advancing to term 76
I20260501 08:11:04.995708 1519 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.995909 1519 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 76 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:04.996165 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 76 candidate_status { last_received { term: 74 index: 79 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:04.996165 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 76 candidate_status { last_received { term: 74 index: 79 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:04.996268 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 75 FOLLOWER]: Advancing to term 76
I20260501 08:11:04.996286 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 75 FOLLOWER]: Advancing to term 76
I20260501 08:11:04.996935 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 76 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 76.
I20260501 08:11:04.996935 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 76 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 76.
I20260501 08:11:04.997195 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 76 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters:
I20260501 08:11:04.997325 1519 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 76 FOLLOWER]: Leader election won for term 76
I20260501 08:11:04.997418 1519 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 76 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:04.997496 1519 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 74.79, Last appended by leader: 79, Current term: 76, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:04.997992 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 74 to 76, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 76 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:05.073340 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 76 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 74 index: 79. Preceding OpId from leader: term: 76 index: 80. (index mismatch)
I20260501 08:11:05.073738 1519 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20260501 08:11:05.075244 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 76 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 74 index: 79. Preceding OpId from leader: term: 76 index: 80. (index mismatch)
I20260501 08:11:05.075498 1519 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20260501 08:11:05.076710 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:05.177032 1528 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 76 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:05.177168 1528 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 76 FOLLOWER]: Advancing to term 77
I20260501 08:11:05.178069 1528 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.178396 1528 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 77 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:05.178480 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 77 candidate_status { last_received { term: 76 index: 80 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:05.178488 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 77 candidate_status { last_received { term: 76 index: 80 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:05.178735 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 77 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:05.178828 1528 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 77 FOLLOWER]: Leader election lost for term 77. Reason: could not achieve majority
I20260501 08:11:05.191764 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 77 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 76. Current term is 77. Ops: []
I20260501 08:11:05.192168 1527 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 76.80, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20260501 08:11:05.192421 1519 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 76 LEADER]: Stepping down as leader of term 76
I20260501 08:11:05.192476 1519 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 76 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:05.192541 1519 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 76.80, Last appended by leader: 80, Current term: 76, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.192629 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 76 FOLLOWER]: Advancing to term 77
I20260501 08:11:05.268874 1504 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 76 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:05.269089 1504 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 76 FOLLOWER]: Advancing to term 77
I20260501 08:11:05.269922 1504 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.270143 1504 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 77 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:05.270399 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 77 candidate_status { last_received { term: 76 index: 80 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:05.270525 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 77 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 77: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:05.270428 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 77 candidate_status { last_received { term: 76 index: 80 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:05.271162 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 77 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 77.
I20260501 08:11:05.271368 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 77 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:05.271462 1504 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 77 FOLLOWER]: Leader election won for term 77
I20260501 08:11:05.271579 1504 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 77 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:05.271623 1504 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 76.80, Last appended by leader: 80, Current term: 77, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.272223 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 76 to 77, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 77 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:05.293987 1528 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 77 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:05.294091 1528 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 77 FOLLOWER]: Advancing to term 78
I20260501 08:11:05.294786 1528 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.294991 1528 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 78 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:05.295218 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 78 candidate_status { last_received { term: 76 index: 80 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:05.295246 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 78 candidate_status { last_received { term: 76 index: 80 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:05.295372 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 77 FOLLOWER]: Advancing to term 78
I20260501 08:11:05.295980 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 78 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 78.
I20260501 08:11:05.296172 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 78 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:05.296303 1528 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 78 FOLLOWER]: Leader election won for term 78
I20260501 08:11:05.296383 1528 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 78 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:05.296443 1528 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 76.80, Last appended by leader: 80, Current term: 78, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.296941 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 77 to 78, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 78 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:05.364584 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 78 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 77. Current term is 78. Ops: []
I20260501 08:11:05.364884 1504 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20260501 08:11:05.365032 1504 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 78.81 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 80
I20260501 08:11:05.365104 1529 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 77 LEADER]: Stepping down as leader of term 77
I20260501 08:11:05.365159 1529 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 77 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:05.365236 1529 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 77.81, Last appended by leader: 81, Current term: 77, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.365336 1529 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 77 FOLLOWER]: Advancing to term 78
I20260501 08:11:05.403815 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 78 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 76 index: 80. Preceding OpId from leader: term: 78 index: 81. (index mismatch)
I20260501 08:11:05.404225 1528 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20260501 08:11:05.405419 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 78 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 77 index: 81. Preceding OpId from leader: term: 78 index: 81. (term mismatch)
I20260501 08:11:05.405479 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 80
I20260501 08:11:05.405499 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 77.81
I20260501 08:11:05.405517 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 78 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:05.405668 1528 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20260501 08:11:05.858332 1519 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 78 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:05.858449 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 78 FOLLOWER]: Advancing to term 79
I20260501 08:11:05.859158 1519 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.859369 1519 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 79 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:05.859643 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 79 candidate_status { last_received { term: 78 index: 81 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:05.859638 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 79 candidate_status { last_received { term: 78 index: 81 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:05.859910 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 79 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:05.860021 1519 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 79 FOLLOWER]: Leader election lost for term 79. Reason: could not achieve majority
I20260501 08:11:05.866815 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 79 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 78. Current term is 79. Ops: []
I20260501 08:11:05.867218 1530 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 78.81, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20260501 08:11:05.867378 1530 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 78 LEADER]: Stepping down as leader of term 78
I20260501 08:11:05.867422 1530 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 78 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:05.867501 1530 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 78.81, Last appended by leader: 81, Current term: 78, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.867595 1530 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 78 FOLLOWER]: Advancing to term 79
I20260501 08:11:05.876019 1529 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 78 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:05.876190 1529 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 78 FOLLOWER]: Advancing to term 79
I20260501 08:11:05.876937 1529 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.877167 1529 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 79 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:05.877396 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 79 candidate_status { last_received { term: 78 index: 81 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:05.877396 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 79 candidate_status { last_received { term: 78 index: 81 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:05.877530 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 79 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 79: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:05.878152 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 79 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 79.
I20260501 08:11:05.878361 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 79 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:05.878456 1529 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 79 FOLLOWER]: Leader election won for term 79
I20260501 08:11:05.878568 1529 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 79 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:05.878611 1529 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 78.81, Last appended by leader: 81, Current term: 79, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.879197 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 78 to 79, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 79 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:05.971479 1519 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 79 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:05.971589 1519 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 79 FOLLOWER]: Advancing to term 80
I20260501 08:11:05.972424 1519 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.972612 1519 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 80 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:05.972860 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 80 candidate_status { last_received { term: 78 index: 81 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:05.972860 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 80 candidate_status { last_received { term: 78 index: 81 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:05.972986 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 79 FOLLOWER]: Advancing to term 80
I20260501 08:11:05.973631 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 80 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 80.
I20260501 08:11:05.973902 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 80 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:05.974066 1519 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 80 FOLLOWER]: Leader election won for term 80
I20260501 08:11:05.974285 1519 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 80 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:05.974365 1519 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 78.81, Last appended by leader: 81, Current term: 80, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.974941 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 79 to 80, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 80 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:05.994059 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 80 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 79. Current term is 80. Ops: []
I20260501 08:11:05.994539 1529 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20260501 08:11:05.994679 1529 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 80.82 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 81
I20260501 08:11:05.994765 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 79 LEADER]: Stepping down as leader of term 79
I20260501 08:11:05.994830 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 79 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:05.994912 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 79.82, Last appended by leader: 82, Current term: 79, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:05.995013 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 79 FOLLOWER]: Advancing to term 80
I20260501 08:11:06.054035 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 80 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 78 index: 81. Preceding OpId from leader: term: 80 index: 82. (index mismatch)
I20260501 08:11:06.054401 1519 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20260501 08:11:06.055771 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 80 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 79 index: 82. Preceding OpId from leader: term: 80 index: 82. (term mismatch)
I20260501 08:11:06.055838 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 81
I20260501 08:11:06.055859 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 79.82
I20260501 08:11:06.055876 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 80 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:06.056084 1519 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20260501 08:11:06.077018 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:06.156795 1530 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 80 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:06.156992 1530 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 80 FOLLOWER]: Advancing to term 81
I20260501 08:11:06.157728 1530 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.157966 1530 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 81 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:06.158274 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 81 candidate_status { last_received { term: 80 index: 82 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:06.158283 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 81 candidate_status { last_received { term: 80 index: 82 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:06.158500 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 81 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:06.158629 1530 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 81 FOLLOWER]: Leader election lost for term 81. Reason: could not achieve majority
I20260501 08:11:06.175485 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 81 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 80. Current term is 81. Ops: []
I20260501 08:11:06.175786 1536 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 80.82, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20260501 08:11:06.175941 1536 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 80 LEADER]: Stepping down as leader of term 80
I20260501 08:11:06.175985 1536 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 80 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:06.176077 1536 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 80.82, Last appended by leader: 82, Current term: 80, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.176178 1536 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 80 FOLLOWER]: Advancing to term 81
I20260501 08:11:06.239594 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 80 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:06.239703 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 80 FOLLOWER]: Advancing to term 81
I20260501 08:11:06.240423 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.240634 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 81 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:06.240864 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 81 candidate_status { last_received { term: 80 index: 82 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:06.240882 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 81 candidate_status { last_received { term: 80 index: 82 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:06.241009 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 81 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 81: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:06.241600 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 81 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 81.
I20260501 08:11:06.241834 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 81 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:06.242000 1531 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 81 FOLLOWER]: Leader election won for term 81
I20260501 08:11:06.242077 1531 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 81 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:06.242125 1531 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 80.82, Last appended by leader: 82, Current term: 81, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.242657 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 80 to 81, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 81 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:06.300292 1530 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 81 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:06.300398 1530 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 81 FOLLOWER]: Advancing to term 82
I20260501 08:11:06.301095 1530 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.301311 1530 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 82 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:06.301540 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 82 candidate_status { last_received { term: 80 index: 82 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:06.301544 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 82 candidate_status { last_received { term: 80 index: 82 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:06.301676 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 81 FOLLOWER]: Advancing to term 82
I20260501 08:11:06.302291 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 82 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 82.
I20260501 08:11:06.302511 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 82 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:06.302620 1530 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 82 FOLLOWER]: Leader election won for term 82
I20260501 08:11:06.302713 1530 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 82 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:06.302767 1530 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 80.82, Last appended by leader: 82, Current term: 82, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.303351 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 81 to 82, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 82 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:06.340142 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 82 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 81. Current term is 82. Ops: []
I20260501 08:11:06.340440 1531 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20260501 08:11:06.340577 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 81 LEADER]: Stepping down as leader of term 81
I20260501 08:11:06.340617 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 81 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:06.340665 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 81.83, Last appended by leader: 83, Current term: 81, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.340723 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 81 FOLLOWER]: Advancing to term 82
I20260501 08:11:06.415279 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 82 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 80 index: 82. Preceding OpId from leader: term: 82 index: 83. (index mismatch)
I20260501 08:11:06.415688 1530 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20260501 08:11:06.417157 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 82 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 81 index: 83. Preceding OpId from leader: term: 82 index: 83. (term mismatch)
I20260501 08:11:06.417222 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 82
I20260501 08:11:06.417248 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 81.83
I20260501 08:11:06.417272 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 82 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:06.417552 1537 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20260501 08:11:06.595258 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 82 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:06.595364 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 82 FOLLOWER]: Advancing to term 83
I20260501 08:11:06.596973 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.597249 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 83 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:06.597599 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 83 candidate_status { last_received { term: 82 index: 83 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:06.597635 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 83 candidate_status { last_received { term: 82 index: 83 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:06.597880 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 83 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:06.597965 1531 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 83 FOLLOWER]: Leader election lost for term 83. Reason: could not achieve majority
I20260501 08:11:06.598057 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 83 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 82. Current term is 83. Ops: []
I20260501 08:11:06.598248 1530 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 82.83, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20260501 08:11:06.598340 1530 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 82 LEADER]: Stepping down as leader of term 82
I20260501 08:11:06.598390 1530 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 82 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:06.598450 1530 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 82.83, Last appended by leader: 83, Current term: 82, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.598529 1530 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 82 FOLLOWER]: Advancing to term 83
I20260501 08:11:06.608343 1536 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 82 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:06.608446 1536 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 82 FOLLOWER]: Advancing to term 83
I20260501 08:11:06.609165 1536 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.609441 1536 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 83 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:06.609658 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 83 candidate_status { last_received { term: 82 index: 83 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:06.609714 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 83 candidate_status { last_received { term: 82 index: 83 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:06.609762 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 83 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 83: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:06.610301 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 83 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 83.
I20260501 08:11:06.610512 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 83 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:06.610603 1536 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 83 FOLLOWER]: Leader election won for term 83
I20260501 08:11:06.610682 1536 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 83 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:06.610739 1536 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 82.83, Last appended by leader: 83, Current term: 83, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.611308 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 82 to 83, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 83 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:06.715588 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 83 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:06.715696 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 83 FOLLOWER]: Advancing to term 84
I20260501 08:11:06.716497 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.716688 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 84 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:06.716987 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 84 candidate_status { last_received { term: 82 index: 83 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:06.716987 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 84 candidate_status { last_received { term: 82 index: 83 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:06.717173 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 83 FOLLOWER]: Advancing to term 84
I20260501 08:11:06.717727 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 84 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 84.
I20260501 08:11:06.717929 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 84 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:06.718060 1531 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 84 FOLLOWER]: Leader election won for term 84
I20260501 08:11:06.718152 1531 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 84 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:06.718209 1531 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 82.83, Last appended by leader: 83, Current term: 84, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.718456 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 84 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 83. Current term is 84. Ops: []
I20260501 08:11:06.718724 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 83 to 84, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 84 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:06.718937 1536 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20260501 08:11:06.719033 1536 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 83 LEADER]: Stepping down as leader of term 83
I20260501 08:11:06.719058 1536 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 83 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:06.719130 1536 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 83.84, Last appended by leader: 84, Current term: 83, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:06.719218 1536 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 83 FOLLOWER]: Advancing to term 84
I20260501 08:11:06.825403 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 84 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 82 index: 83. Preceding OpId from leader: term: 84 index: 84. (index mismatch)
I20260501 08:11:06.825405 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 84 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 83 index: 84. Preceding OpId from leader: term: 84 index: 84. (term mismatch)
I20260501 08:11:06.825549 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 83
I20260501 08:11:06.825570 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 83.84
I20260501 08:11:06.825589 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 84 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:06.825709 1531 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20260501 08:11:06.825781 1529 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20260501 08:11:07.018760 1530 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 84 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:07.018891 1530 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 84 FOLLOWER]: Advancing to term 85
I20260501 08:11:07.019606 1530 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.019822 1530 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 85 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:07.020046 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 85 candidate_status { last_received { term: 84 index: 84 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:07.020057 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 85 candidate_status { last_received { term: 84 index: 84 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:07.020332 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 85 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:07.020514 1530 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 85 FOLLOWER]: Leader election lost for term 85. Reason: could not achieve majority
I20260501 08:11:07.038010 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 85 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 84. Current term is 85. Ops: []
I20260501 08:11:07.038311 1538 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 84.84, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20260501 08:11:07.038434 1538 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 84 LEADER]: Stepping down as leader of term 84
I20260501 08:11:07.038455 1538 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 84 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:07.038546 1538 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 84.84, Last appended by leader: 84, Current term: 84, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.038637 1538 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 84 FOLLOWER]: Advancing to term 85
I20260501 08:11:07.077268 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:07.112627 1536 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 84 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:07.112735 1536 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 84 FOLLOWER]: Advancing to term 85
I20260501 08:11:07.113435 1536 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.113735 1536 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 85 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:07.113873 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 85 candidate_status { last_received { term: 84 index: 84 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:07.113921 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 85 candidate_status { last_received { term: 84 index: 84 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:07.114082 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 85 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 85: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:07.114702 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 85 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 85.
I20260501 08:11:07.114965 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 85 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:07.115079 1536 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 85 FOLLOWER]: Leader election won for term 85
I20260501 08:11:07.115159 1536 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 85 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:07.115231 1536 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 84.84, Last appended by leader: 84, Current term: 85, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.115746 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 84 to 85, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 85 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:07.161065 1530 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 85 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:07.161178 1530 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 85 FOLLOWER]: Advancing to term 86
I20260501 08:11:07.162032 1530 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.162230 1530 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 86 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:07.162509 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 86 candidate_status { last_received { term: 84 index: 84 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:07.162509 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 86 candidate_status { last_received { term: 84 index: 84 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:07.162657 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 85 FOLLOWER]: Advancing to term 86
I20260501 08:11:07.163259 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 86 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 86.
I20260501 08:11:07.163477 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 86 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:07.163619 1530 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 86 FOLLOWER]: Leader election won for term 86
I20260501 08:11:07.163769 1530 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 86 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:07.163838 1530 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 84.84, Last appended by leader: 84, Current term: 86, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.164268 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 85 to 86, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 86 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:07.200502 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 86 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 85. Current term is 86. Ops: []
I20260501 08:11:07.200956 1536 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20260501 08:11:07.201107 1536 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 85 LEADER]: Stepping down as leader of term 85
I20260501 08:11:07.201148 1536 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 85 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:07.201208 1536 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 85.85, Last appended by leader: 85, Current term: 85, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.201263 1536 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 85 FOLLOWER]: Advancing to term 86
I20260501 08:11:07.240729 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 86 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 85 index: 85. Preceding OpId from leader: term: 86 index: 85. (term mismatch)
I20260501 08:11:07.240811 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 84
I20260501 08:11:07.240826 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 85.85
I20260501 08:11:07.240840 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 86 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:07.241108 1530 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20260501 08:11:07.242234 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 86 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 84 index: 84. Preceding OpId from leader: term: 86 index: 85. (index mismatch)
I20260501 08:11:07.242594 1530 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20260501 08:11:07.429912 1536 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 86 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:07.430079 1536 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 86 FOLLOWER]: Advancing to term 87
I20260501 08:11:07.430982 1536 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.431226 1536 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 87 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:07.431535 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 87 candidate_status { last_received { term: 86 index: 85 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:07.431535 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 87 candidate_status { last_received { term: 86 index: 85 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:07.432027 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 87 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:07.432224 1536 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 87 FOLLOWER]: Leader election lost for term 87. Reason: could not achieve majority
I20260501 08:11:07.442299 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 87 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 86. Current term is 87. Ops: []
I20260501 08:11:07.442656 1539 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 86.85, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20260501 08:11:07.442826 1539 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 86 LEADER]: Stepping down as leader of term 86
I20260501 08:11:07.442881 1539 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 86 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:07.442960 1539 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 86.85, Last appended by leader: 85, Current term: 86, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.443066 1539 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 86 FOLLOWER]: Advancing to term 87
I20260501 08:11:07.505890 1538 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 86 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:07.506060 1538 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 86 FOLLOWER]: Advancing to term 87
I20260501 08:11:07.506767 1538 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.507009 1538 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 87 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:07.507241 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 87 candidate_status { last_received { term: 86 index: 85 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:07.507261 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 87 candidate_status { last_received { term: 86 index: 85 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:07.507395 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 87: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:07.508013 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 87 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 87.
I20260501 08:11:07.508291 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 87 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:07.508410 1538 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 87 FOLLOWER]: Leader election won for term 87
I20260501 08:11:07.508491 1538 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 87 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:07.508556 1538 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 86.85, Last appended by leader: 85, Current term: 87, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.509085 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 86 to 87, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 87 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:07.545929 1536 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 87 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:07.546041 1536 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 87 FOLLOWER]: Advancing to term 88
I20260501 08:11:07.546706 1536 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.546908 1536 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 88 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:07.547173 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 88 candidate_status { last_received { term: 86 index: 85 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:07.547173 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 88 candidate_status { last_received { term: 86 index: 85 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:07.547295 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 87 FOLLOWER]: Advancing to term 88
I20260501 08:11:07.547909 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 88 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 88.
I20260501 08:11:07.548216 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 88 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:07.548313 1536 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 88 FOLLOWER]: Leader election won for term 88
I20260501 08:11:07.548390 1536 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 88 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:07.548466 1536 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 86.85, Last appended by leader: 85, Current term: 88, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.549006 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 87 to 88, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 88 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:07.585793 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 88 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 87. Current term is 88. Ops: []
I20260501 08:11:07.586115 1538 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20260501 08:11:07.586278 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 87 LEADER]: Stepping down as leader of term 87
I20260501 08:11:07.586330 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 87 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:07.586393 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 87.86, Last appended by leader: 86, Current term: 87, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.586505 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 87 FOLLOWER]: Advancing to term 88
I20260501 08:11:07.640990 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 88 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 87 index: 86. Preceding OpId from leader: term: 88 index: 86. (term mismatch)
I20260501 08:11:07.641079 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 85
I20260501 08:11:07.641098 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 87.86
I20260501 08:11:07.641113 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 88 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:07.641390 1536 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20260501 08:11:07.642545 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 88 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 86 index: 85. Preceding OpId from leader: term: 88 index: 86. (index mismatch)
I20260501 08:11:07.642717 1541 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20260501 08:11:07.744047 1540 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 88 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:07.744185 1540 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 88 FOLLOWER]: Advancing to term 89
I20260501 08:11:07.744953 1540 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.745150 1540 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 89 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:07.745471 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 89 candidate_status { last_received { term: 88 index: 86 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:07.745491 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 89 candidate_status { last_received { term: 88 index: 86 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:07.745735 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 89 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:07.745838 1540 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 89 FOLLOWER]: Leader election lost for term 89. Reason: could not achieve majority
I20260501 08:11:07.764416 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 89 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 88. Current term is 89. Ops: []
I20260501 08:11:07.764755 1536 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 88.86, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20260501 08:11:07.764875 1536 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 88 LEADER]: Stepping down as leader of term 88
I20260501 08:11:07.764894 1536 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 88 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:07.764943 1536 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 88.86, Last appended by leader: 86, Current term: 88, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.765040 1536 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 88 FOLLOWER]: Advancing to term 89
I20260501 08:11:07.818518 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 88 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:07.818754 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 88 FOLLOWER]: Advancing to term 89
I20260501 08:11:07.819482 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.819684 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 89 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:07.819911 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 89 candidate_status { last_received { term: 88 index: 86 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:07.819927 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 89 candidate_status { last_received { term: 88 index: 86 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:07.820007 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 89 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 89: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:07.820585 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 89 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 89.
I20260501 08:11:07.820788 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 89 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters:
I20260501 08:11:07.821149 1531 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 89 FOLLOWER]: Leader election won for term 89
I20260501 08:11:07.821241 1531 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 89 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:07.821318 1531 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 88.86, Last appended by leader: 86, Current term: 89, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.821794 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 88 to 89, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 89 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:07.875218 1540 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 89 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:07.875337 1540 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 89 FOLLOWER]: Advancing to term 90
I20260501 08:11:07.876253 1540 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.876457 1540 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 90 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:07.876756 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 90 candidate_status { last_received { term: 88 index: 86 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:07.876823 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 89 FOLLOWER]: Advancing to term 90
I20260501 08:11:07.876763 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 90 candidate_status { last_received { term: 88 index: 86 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:07.877549 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 90 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 90.
I20260501 08:11:07.877753 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 90 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:07.877843 1540 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 90 FOLLOWER]: Leader election won for term 90
I20260501 08:11:07.877888 1540 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 90 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:07.877938 1540 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 88.86, Last appended by leader: 86, Current term: 90, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.878564 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 89 to 90, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 90 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:07.932560 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 90 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 89. Current term is 90. Ops: []
I20260501 08:11:07.932723 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 90 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 89. Current term is 90. Ops: []
I20260501 08:11:07.932857 1531 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20260501 08:11:07.933136 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 89 LEADER]: Stepping down as leader of term 89
I20260501 08:11:07.933187 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 89 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:07.933152 1538 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20260501 08:11:07.933344 1538 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 90.87 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 86
I20260501 08:11:07.933422 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 89.87, Last appended by leader: 87, Current term: 89, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:07.933517 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 89 FOLLOWER]: Advancing to term 90
I20260501 08:11:07.979736 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 90 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 89 index: 87. Preceding OpId from leader: term: 90 index: 87. (term mismatch)
I20260501 08:11:07.979825 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 86
I20260501 08:11:07.979841 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 89.87
I20260501 08:11:07.979856 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 90 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:07.980147 1540 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20260501 08:11:07.981525 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 90 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 88 index: 86. Preceding OpId from leader: term: 90 index: 87. (index mismatch)
I20260501 08:11:07.981781 1539 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20260501 08:11:08.077582 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:08.161685 1536 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 90 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:08.161828 1536 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 90 FOLLOWER]: Advancing to term 91
I20260501 08:11:08.162489 1536 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.162706 1536 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 91 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:08.163012 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 91 candidate_status { last_received { term: 90 index: 87 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:08.163012 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 91 candidate_status { last_received { term: 90 index: 87 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:08.163326 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 91 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:08.163444 1536 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 91 FOLLOWER]: Leader election lost for term 91. Reason: could not achieve majority
I20260501 08:11:08.167279 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 91 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 90. Current term is 91. Ops: []
I20260501 08:11:08.167594 1539 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 90.87, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20260501 08:11:08.167747 1539 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 90 LEADER]: Stepping down as leader of term 90
I20260501 08:11:08.167780 1539 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 90 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:08.167838 1539 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 90.87, Last appended by leader: 87, Current term: 90, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.167932 1539 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 90 FOLLOWER]: Advancing to term 91
I20260501 08:11:08.194933 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 90 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:08.195046 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 90 FOLLOWER]: Advancing to term 91
I20260501 08:11:08.195727 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.195920 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 91 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:08.196229 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 91 candidate_status { last_received { term: 90 index: 87 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:08.196305 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 91 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 91: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:08.196233 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 91 candidate_status { last_received { term: 90 index: 87 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:08.196931 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 91 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 91.
I20260501 08:11:08.197230 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 91 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:08.197355 1531 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 91 FOLLOWER]: Leader election won for term 91
I20260501 08:11:08.197412 1531 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 91 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:08.197458 1531 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 90.87, Last appended by leader: 87, Current term: 91, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.198130 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 90 to 91, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 91 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:08.270938 1536 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 91 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:08.271086 1536 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 91 FOLLOWER]: Advancing to term 92
I20260501 08:11:08.271747 1536 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.271953 1536 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 92 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:08.272253 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 92 candidate_status { last_received { term: 90 index: 87 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:08.272284 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 92 candidate_status { last_received { term: 90 index: 87 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:08.272336 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 91 FOLLOWER]: Advancing to term 92
I20260501 08:11:08.272910 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 92 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 92.
I20260501 08:11:08.273095 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 92 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:08.273245 1536 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 92 FOLLOWER]: Leader election won for term 92
I20260501 08:11:08.273308 1536 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 92 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:08.273352 1536 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 90.87, Last appended by leader: 87, Current term: 92, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.273945 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 91 to 92, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 92 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:08.277441 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 92 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 91. Current term is 92. Ops: []
I20260501 08:11:08.277722 1531 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20260501 08:11:08.277878 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 91 LEADER]: Stepping down as leader of term 91
I20260501 08:11:08.277920 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 91 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:08.277987 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 91.88, Last appended by leader: 88, Current term: 91, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.278076 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 91 FOLLOWER]: Advancing to term 92
I20260501 08:11:08.352559 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 92 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 91 index: 88. Preceding OpId from leader: term: 92 index: 88. (term mismatch)
I20260501 08:11:08.352644 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 87
I20260501 08:11:08.352661 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 91.88
I20260501 08:11:08.352676 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 92 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:08.352910 1536 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20260501 08:11:08.354087 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 92 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 90 index: 87. Preceding OpId from leader: term: 92 index: 88. (index mismatch)
I20260501 08:11:08.354300 1536 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20260501 08:11:08.541942 1543 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 92 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:08.542061 1543 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 92 FOLLOWER]: Advancing to term 93
I20260501 08:11:08.542805 1543 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.543013 1543 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 93 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:08.543232 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 93 candidate_status { last_received { term: 92 index: 88 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:08.543264 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 93 candidate_status { last_received { term: 92 index: 88 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:08.543563 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 93 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:08.543658 1543 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 93 FOLLOWER]: Leader election lost for term 93. Reason: could not achieve majority
I20260501 08:11:08.550524 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 92 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:08.550628 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 92 FOLLOWER]: Advancing to term 93
I20260501 08:11:08.551272 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.551468 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 93 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:08.551723 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 93 candidate_status { last_received { term: 92 index: 88 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:08.551723 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 93 candidate_status { last_received { term: 92 index: 88 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:08.551863 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 93 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 93: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:08.552033 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 93 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:08.552140 1531 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 93 FOLLOWER]: Leader election lost for term 93. Reason: could not achieve majority
I20260501 08:11:08.552269 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 93 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 92. Current term is 93. Ops: []
I20260501 08:11:08.552481 1544 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 92.88, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20260501 08:11:08.552636 1544 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 92 LEADER]: Stepping down as leader of term 92
I20260501 08:11:08.552692 1544 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 92 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:08.552779 1544 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 92.88, Last appended by leader: 88, Current term: 92, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.552875 1544 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 92 FOLLOWER]: Advancing to term 93
I20260501 08:11:08.653831 1543 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 93 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:08.653951 1543 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 93 FOLLOWER]: Advancing to term 94
I20260501 08:11:08.654693 1543 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.654996 1543 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 94 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:08.655123 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 94 candidate_status { last_received { term: 92 index: 88 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:08.655195 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 94 candidate_status { last_received { term: 92 index: 88 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:08.655201 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 93 FOLLOWER]: Advancing to term 94
I20260501 08:11:08.655249 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 93 FOLLOWER]: Advancing to term 94
I20260501 08:11:08.655911 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 94 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 94.
I20260501 08:11:08.655911 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 94 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 94.
I20260501 08:11:08.656194 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 94 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters:
I20260501 08:11:08.656342 1543 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 94 FOLLOWER]: Leader election won for term 94
I20260501 08:11:08.656416 1543 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 94 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:08.656473 1543 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 92.88, Last appended by leader: 88, Current term: 94, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:08.656955 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 92 to 94, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 94 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:08.748636 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 94 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 92 index: 88. Preceding OpId from leader: term: 94 index: 89. (index mismatch)
I20260501 08:11:08.749056 1543 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20260501 08:11:08.750378 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 94 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 92 index: 88. Preceding OpId from leader: term: 94 index: 89. (index mismatch)
I20260501 08:11:08.750568 1543 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20260501 08:11:09.022531 1538 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 94 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:09.022642 1538 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 94 FOLLOWER]: Advancing to term 95
I20260501 08:11:09.023319 1538 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.023510 1538 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 95 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:09.023756 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 95 candidate_status { last_received { term: 94 index: 89 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:09.023900 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 95 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 94. Current term is 95. Ops: []
I20260501 08:11:09.023754 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 95 candidate_status { last_received { term: 94 index: 89 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:09.024169 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 95 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:09.024224 1545 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 94.89, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20260501 08:11:09.024261 1538 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 95 FOLLOWER]: Leader election lost for term 95. Reason: could not achieve majority
I20260501 08:11:09.024327 1539 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 94 LEADER]: Stepping down as leader of term 94
I20260501 08:11:09.024372 1539 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 94 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:09.024434 1539 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 94.89, Last appended by leader: 89, Current term: 94, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.024549 1539 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 94 FOLLOWER]: Advancing to term 95
I20260501 08:11:09.077890 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:09.106664 1544 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 94 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:09.106778 1544 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 94 FOLLOWER]: Advancing to term 95
I20260501 08:11:09.107590 1544 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.107786 1544 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 95 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:09.108035 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 95 candidate_status { last_received { term: 94 index: 89 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:09.108074 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 95 candidate_status { last_received { term: 94 index: 89 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:09.108238 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 95 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 95: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:09.108815 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 95.
I20260501 08:11:09.109078 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 95 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:09.109215 1544 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 95 FOLLOWER]: Leader election won for term 95
I20260501 08:11:09.109373 1544 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 95 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:09.109465 1544 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 94.89, Last appended by leader: 89, Current term: 95, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.110045 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 94 to 95, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 95 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:09.170797 1538 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 95 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:09.171007 1538 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 95 FOLLOWER]: Advancing to term 96
I20260501 08:11:09.172070 1538 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.172300 1538 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 96 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:09.172493 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 96 candidate_status { last_received { term: 94 index: 89 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:09.172528 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 96 candidate_status { last_received { term: 94 index: 89 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:09.172580 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 95 FOLLOWER]: Advancing to term 96
I20260501 08:11:09.173524 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 96 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 96.
I20260501 08:11:09.173795 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 96 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:09.173911 1538 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 96 FOLLOWER]: Leader election won for term 96
I20260501 08:11:09.173971 1538 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 96 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:09.174045 1538 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 94.89, Last appended by leader: 89, Current term: 96, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.174597 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 95 to 96, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 96 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:09.202998 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 96 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 95. Current term is 96. Ops: []
I20260501 08:11:09.203403 1544 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20260501 08:11:09.203567 1544 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 95 LEADER]: Stepping down as leader of term 95
I20260501 08:11:09.203627 1544 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 95 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:09.203703 1544 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 95.90, Last appended by leader: 90, Current term: 95, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.203795 1544 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 95 FOLLOWER]: Advancing to term 96
I20260501 08:11:09.268335 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 96 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 95 index: 90. Preceding OpId from leader: term: 96 index: 90. (term mismatch)
I20260501 08:11:09.268451 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 89
I20260501 08:11:09.268473 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 95.90
I20260501 08:11:09.268497 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 96 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:09.268800 1538 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20260501 08:11:09.270135 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 96 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 94 index: 89. Preceding OpId from leader: term: 96 index: 90. (index mismatch)
I20260501 08:11:09.270401 1547 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20260501 08:11:09.371605 1545 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 96 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:09.371717 1545 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 96 FOLLOWER]: Advancing to term 97
I20260501 08:11:09.372555 1545 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.372749 1545 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 97 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:09.373000 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 97 candidate_status { last_received { term: 96 index: 90 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:09.373000 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 97 candidate_status { last_received { term: 96 index: 90 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:09.373268 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 97 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:09.373410 1545 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 97 FOLLOWER]: Leader election lost for term 97. Reason: could not achieve majority
I20260501 08:11:09.383852 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 97 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 96. Current term is 97. Ops: []
I20260501 08:11:09.384245 1531 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 96.90, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20260501 08:11:09.384375 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 96 LEADER]: Stepping down as leader of term 96
I20260501 08:11:09.384398 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 96 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:09.384465 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 96.90, Last appended by leader: 90, Current term: 96, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.384555 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 96 FOLLOWER]: Advancing to term 97
I20260501 08:11:09.463454 1544 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 96 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:09.463557 1544 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 96 FOLLOWER]: Advancing to term 97
I20260501 08:11:09.464323 1544 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.464601 1544 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 97 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:09.464722 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 97 candidate_status { last_received { term: 96 index: 90 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:09.464722 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 97 candidate_status { last_received { term: 96 index: 90 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:09.464860 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 97 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 97: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:09.465497 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 97 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 97.
I20260501 08:11:09.465695 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 97 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:09.465792 1544 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 97 FOLLOWER]: Leader election won for term 97
I20260501 08:11:09.465847 1544 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 97 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:09.465907 1544 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 96.90, Last appended by leader: 90, Current term: 97, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.466367 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 96 to 97, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 97 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:09.483260 1545 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 97 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:09.483388 1545 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 97 FOLLOWER]: Advancing to term 98
I20260501 08:11:09.484198 1545 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.484397 1545 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 98 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:09.484647 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 98 candidate_status { last_received { term: 96 index: 90 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:09.484634 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 98 candidate_status { last_received { term: 96 index: 90 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:09.484771 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 97 FOLLOWER]: Advancing to term 98
I20260501 08:11:09.485440 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 98 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 98.
I20260501 08:11:09.485683 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 98 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:09.485782 1545 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 98 FOLLOWER]: Leader election won for term 98
I20260501 08:11:09.485850 1545 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 98 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:09.485926 1545 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 96.90, Last appended by leader: 90, Current term: 98, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.486449 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 97 to 98, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 98 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:09.559031 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 98 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 97. Current term is 98. Ops: []
I20260501 08:11:09.559510 1546 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20260501 08:11:09.559715 1546 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 97 LEADER]: Stepping down as leader of term 97
I20260501 08:11:09.559751 1546 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 97 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:09.559832 1546 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 97.91, Last appended by leader: 91, Current term: 97, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.559916 1546 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 97 FOLLOWER]: Advancing to term 98
I20260501 08:11:09.576269 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 98 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 96 index: 90. Preceding OpId from leader: term: 98 index: 91. (index mismatch)
I20260501 08:11:09.576658 1545 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20260501 08:11:09.577883 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 98 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 97 index: 91. Preceding OpId from leader: term: 98 index: 91. (term mismatch)
I20260501 08:11:09.577947 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 90
I20260501 08:11:09.577970 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 97.91
I20260501 08:11:09.577987 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 98 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:09.578213 1539 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20260501 08:11:09.763330 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 98 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:09.763437 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 98 FOLLOWER]: Advancing to term 99
I20260501 08:11:09.764267 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.764452 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 99 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:09.764705 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 99 candidate_status { last_received { term: 98 index: 91 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:09.764724 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 99 candidate_status { last_received { term: 98 index: 91 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:09.765005 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 99 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:09.765152 1531 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 99 FOLLOWER]: Leader election lost for term 99. Reason: could not achieve majority
I20260501 08:11:09.772281 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 99 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 98. Current term is 99. Ops: []
I20260501 08:11:09.772588 1548 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 98.91, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20260501 08:11:09.772723 1548 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 98 LEADER]: Stepping down as leader of term 98
I20260501 08:11:09.772742 1548 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 98 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:09.772840 1548 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 98.91, Last appended by leader: 91, Current term: 98, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.772928 1548 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 98 FOLLOWER]: Advancing to term 99
I20260501 08:11:09.785625 1544 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 98 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:09.785728 1544 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 98 FOLLOWER]: Advancing to term 99
I20260501 08:11:09.786568 1544 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.786767 1544 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 99 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:09.787029 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 99 candidate_status { last_received { term: 98 index: 91 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:09.787051 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 99 candidate_status { last_received { term: 98 index: 91 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:09.787150 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 99: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:09.787731 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 99 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 99.
I20260501 08:11:09.787945 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 99 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:09.788040 1544 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 99 FOLLOWER]: Leader election won for term 99
I20260501 08:11:09.788093 1544 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 99 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:09.788172 1544 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 98.91, Last appended by leader: 91, Current term: 99, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.788717 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 98 to 99, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 99 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:09.876888 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 99 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 98 index: 91. Preceding OpId from leader: term: 99 index: 92. (index mismatch)
I20260501 08:11:09.877241 1544 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20260501 08:11:09.878340 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 99 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 98 index: 91. Preceding OpId from leader: term: 99 index: 92. (index mismatch)
I20260501 08:11:09.878791 1546 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20260501 08:11:09.978622 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 99 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:09.978732 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 99 FOLLOWER]: Advancing to term 100
I20260501 08:11:09.979434 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.979627 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 100 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:09.979835 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 100 candidate_status { last_received { term: 99 index: 92 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:09.979839 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 100 candidate_status { last_received { term: 99 index: 92 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:09.980312 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 100 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:09.980473 1531 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 100 FOLLOWER]: Leader election lost for term 100. Reason: could not achieve majority
I20260501 08:11:09.998732 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 100 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 99. Current term is 100. Ops: []
I20260501 08:11:09.999059 1546 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 99.92, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20260501 08:11:09.999207 1546 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 99 LEADER]: Stepping down as leader of term 99
I20260501 08:11:09.999234 1546 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 99 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:09.999320 1546 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 99.92, Last appended by leader: 92, Current term: 99, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:09.999413 1546 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 99 FOLLOWER]: Advancing to term 100
I20260501 08:11:10.069645 1548 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 99 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:10.069761 1548 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 99 FOLLOWER]: Advancing to term 100
I20260501 08:11:10.070497 1548 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.070705 1548 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 100 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:10.070940 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 100 candidate_status { last_received { term: 99 index: 92 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:10.070943 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 100 candidate_status { last_received { term: 99 index: 92 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:10.071110 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 100 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 100: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:10.071763 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 100 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 100.
I20260501 08:11:10.071934 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 100 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:10.072031 1548 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 100 FOLLOWER]: Leader election won for term 100
I20260501 08:11:10.072083 1548 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 100 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:10.072157 1548 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 99.92, Last appended by leader: 92, Current term: 100, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.072677 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 99 to 100, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 100 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:10.082060 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 100 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:10.082201 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 100 FOLLOWER]: Advancing to term 101
I20260501 08:11:10.082939 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.083163 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 101 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:10.083369 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 101 candidate_status { last_received { term: 99 index: 92 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:10.083379 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 101 candidate_status { last_received { term: 99 index: 92 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:10.083577 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 100 FOLLOWER]: Advancing to term 101
I20260501 08:11:10.084230 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 101 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 101.
I20260501 08:11:10.084419 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 101 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:10.084517 1531 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 101 FOLLOWER]: Leader election won for term 101
I20260501 08:11:10.084625 1531 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 101 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:10.084677 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:10.084699 1531 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 99.92, Last appended by leader: 92, Current term: 101, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.085206 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 100 to 101, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 101 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:10.161177 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 101 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 100. Current term is 101. Ops: []
I20260501 08:11:10.161593 1548 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20260501 08:11:10.161726 1548 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 100 LEADER]: Stepping down as leader of term 100
I20260501 08:11:10.161767 1548 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 100 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:10.161835 1548 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 100.93, Last appended by leader: 93, Current term: 100, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.161909 1548 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 100 FOLLOWER]: Advancing to term 101
I20260501 08:11:10.201731 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 101 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 100 index: 93. Preceding OpId from leader: term: 101 index: 93. (term mismatch)
I20260501 08:11:10.201813 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 92
I20260501 08:11:10.201828 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 100.93
I20260501 08:11:10.201843 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 101 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:10.202112 1531 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20260501 08:11:10.203400 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 101 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 99 index: 92. Preceding OpId from leader: term: 101 index: 93. (index mismatch)
I20260501 08:11:10.203639 1531 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20260501 08:11:10.304867 1546 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 101 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:10.304973 1546 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 101 FOLLOWER]: Advancing to term 102
I20260501 08:11:10.306020 1546 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.306166 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 102 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 101. Current term is 102. Ops: []
I20260501 08:11:10.306218 1546 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 102 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:10.306417 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 102 candidate_status { last_received { term: 101 index: 93 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:10.306411 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 102 candidate_status { last_received { term: 101 index: 93 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:10.306453 1531 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 101.93, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20260501 08:11:10.306605 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 101 FOLLOWER]: Advancing to term 102
I20260501 08:11:10.306658 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 101 LEADER]: Stepping down as leader of term 101
I20260501 08:11:10.306692 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 101 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:10.306763 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 101.93, Last appended by leader: 93, Current term: 101, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.306862 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 101 FOLLOWER]: Advancing to term 102
I20260501 08:11:10.307351 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 102 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 102.
I20260501 08:11:10.307556 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 102 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:10.307675 1546 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 102 FOLLOWER]: Leader election won for term 102
I20260501 08:11:10.307739 1546 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 102 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:10.307787 1546 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 101.93, Last appended by leader: 93, Current term: 102, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.308373 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 101 to 102, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 102 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:10.409088 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 102 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 101 index: 93. Preceding OpId from leader: term: 102 index: 94. (index mismatch)
I20260501 08:11:10.409513 1546 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20260501 08:11:10.410727 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 102 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 101 index: 93. Preceding OpId from leader: term: 102 index: 94. (index mismatch)
I20260501 08:11:10.411029 1549 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20260501 08:11:10.764748 1548 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 102 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:10.764856 1548 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 102 FOLLOWER]: Advancing to term 103
I20260501 08:11:10.765600 1548 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.765838 1548 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 103 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:10.766065 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 103 candidate_status { last_received { term: 102 index: 94 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:10.766074 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 103 candidate_status { last_received { term: 102 index: 94 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:10.766371 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 103 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:10.766465 1548 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 103 FOLLOWER]: Leader election lost for term 103. Reason: could not achieve majority
I20260501 08:11:10.789130 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 103 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 102. Current term is 103. Ops: []
I20260501 08:11:10.789495 1549 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 102.94, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20260501 08:11:10.789685 1544 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 102 LEADER]: Stepping down as leader of term 102
I20260501 08:11:10.789741 1544 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 102 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:10.789829 1544 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 102.94, Last appended by leader: 94, Current term: 102, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.789974 1544 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 102 FOLLOWER]: Advancing to term 103
I20260501 08:11:10.799489 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 102 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:10.799598 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 102 FOLLOWER]: Advancing to term 103
I20260501 08:11:10.800307 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.800522 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 103 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:10.800748 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 103 candidate_status { last_received { term: 102 index: 94 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:10.800792 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 103 candidate_status { last_received { term: 102 index: 94 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:10.800863 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 103 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 103: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:10.801488 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 103 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 103.
I20260501 08:11:10.801692 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 103 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:10.801795 1531 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 103 FOLLOWER]: Leader election won for term 103
I20260501 08:11:10.801846 1531 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 103 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:10.801903 1531 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 102.94, Last appended by leader: 94, Current term: 103, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.802367 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 102 to 103, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 103 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:10.871104 1548 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 103 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:10.871222 1548 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 103 FOLLOWER]: Advancing to term 104
I20260501 08:11:10.871906 1548 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.872165 1548 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 104 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:10.872398 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 104 candidate_status { last_received { term: 102 index: 94 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:10.872476 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 103 FOLLOWER]: Advancing to term 104
I20260501 08:11:10.872360 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 104 candidate_status { last_received { term: 102 index: 94 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:10.873099 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 104 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 104.
I20260501 08:11:10.873327 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 104 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:10.873430 1548 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 104 FOLLOWER]: Leader election won for term 104
I20260501 08:11:10.873551 1548 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 104 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:10.873594 1548 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 102.94, Last appended by leader: 94, Current term: 104, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.874083 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 103 to 104, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 104 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:10.884939 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 104 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 103. Current term is 104. Ops: []
I20260501 08:11:10.885293 1531 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20260501 08:11:10.885428 1551 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 103 LEADER]: Stepping down as leader of term 103
I20260501 08:11:10.885473 1551 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 103 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:10.885553 1551 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 103.95, Last appended by leader: 95, Current term: 103, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:10.885641 1551 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 103 FOLLOWER]: Advancing to term 104
I20260501 08:11:10.960582 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 104 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 102 index: 94. Preceding OpId from leader: term: 104 index: 95. (index mismatch)
I20260501 08:11:10.960958 1548 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20260501 08:11:10.962218 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 104 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 103 index: 95. Preceding OpId from leader: term: 104 index: 95. (term mismatch)
I20260501 08:11:10.962285 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 94
I20260501 08:11:10.962311 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 103.95
I20260501 08:11:10.962329 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 104 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:10.962571 1553 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20260501 08:11:11.062562 1544 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 104 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:11.062674 1544 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 104 FOLLOWER]: Advancing to term 105
I20260501 08:11:11.063769 1544 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.063987 1544 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 105 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:11.064217 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 105 candidate_status { last_received { term: 104 index: 95 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:11.064285 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 105 candidate_status { last_received { term: 104 index: 95 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:11.064503 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 105 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:11.064606 1544 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 105 FOLLOWER]: Leader election lost for term 105. Reason: could not achieve majority
I20260501 08:11:11.078411 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 105 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 104. Current term is 105. Ops: []
I20260501 08:11:11.078785 1548 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 104.95, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20260501 08:11:11.079101 1552 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 104 LEADER]: Stepping down as leader of term 104
I20260501 08:11:11.079164 1552 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 104 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:11.079241 1552 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 104.95, Last appended by leader: 95, Current term: 104, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.079330 1552 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 104 FOLLOWER]: Advancing to term 105
I20260501 08:11:11.084915 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:11.155611 1551 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 104 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:11.155714 1551 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 104 FOLLOWER]: Advancing to term 105
I20260501 08:11:11.156546 1551 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.156739 1551 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 105 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:11.156946 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 105 candidate_status { last_received { term: 104 index: 95 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:11.157482 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 105 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 105.
I20260501 08:11:11.157620 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 105 candidate_status { last_received { term: 104 index: 95 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:11.157697 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 105 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 105: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:11.157721 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 105 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters:
I20260501 08:11:11.157879 1551 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 105 FOLLOWER]: Leader election won for term 105
I20260501 08:11:11.157940 1551 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 105 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:11.157995 1551 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 104.95, Last appended by leader: 95, Current term: 105, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.158555 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 104 to 105, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 105 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:11.192273 1544 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 105 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:11.192379 1544 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 105 FOLLOWER]: Advancing to term 106
I20260501 08:11:11.193205 1544 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 106 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.193396 1544 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 106 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:11.193614 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 106 candidate_status { last_received { term: 104 index: 95 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:11.193742 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 106 candidate_status { last_received { term: 104 index: 95 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:11.193866 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 105 FOLLOWER]: Advancing to term 106
I20260501 08:11:11.194447 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 106 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 106.
I20260501 08:11:11.194654 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 106 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:11.194766 1544 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 106 FOLLOWER]: Leader election won for term 106
I20260501 08:11:11.194852 1544 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 106 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:11.194908 1544 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 104.95, Last appended by leader: 95, Current term: 106, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.195565 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 105 to 106, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 106 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:11.270920 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 106 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 105. Current term is 106. Ops: []
I20260501 08:11:11.271253 1551 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20260501 08:11:11.271418 1551 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 106.96 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 95
I20260501 08:11:11.271478 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 105 LEADER]: Stepping down as leader of term 105
I20260501 08:11:11.271559 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 105 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:11.271624 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 105.96, Last appended by leader: 96, Current term: 105, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.271718 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 105 FOLLOWER]: Advancing to term 106
I20260501 08:11:11.273837 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 106 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 104 index: 95. Preceding OpId from leader: term: 106 index: 96. (index mismatch)
I20260501 08:11:11.274107 1544 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20260501 08:11:11.275213 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 106 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 105 index: 96. Preceding OpId from leader: term: 106 index: 96. (term mismatch)
I20260501 08:11:11.275264 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 95
I20260501 08:11:11.275285 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 105.96
I20260501 08:11:11.275303 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 106 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:11.275483 1554 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20260501 08:11:11.542848 1552 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 106 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:11.543002 1552 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 106 FOLLOWER]: Advancing to term 107
I20260501 08:11:11.543743 1552 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.543952 1552 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 107 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:11.544171 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 107 candidate_status { last_received { term: 106 index: 96 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:11.544305 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 107 candidate_status { last_received { term: 106 index: 96 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:11.544469 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 107 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:11.544631 1552 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 107 FOLLOWER]: Leader election lost for term 107. Reason: could not achieve majority
I20260501 08:11:11.566550 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 107 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 106. Current term is 107. Ops: []
I20260501 08:11:11.566856 1544 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 106.96, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20260501 08:11:11.567049 1549 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 106 LEADER]: Stepping down as leader of term 106
I20260501 08:11:11.567095 1549 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 106 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:11.567162 1549 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 106.96, Last appended by leader: 96, Current term: 106, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.567261 1549 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 106 FOLLOWER]: Advancing to term 107
I20260501 08:11:11.582628 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 106 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:11.582741 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 106 FOLLOWER]: Advancing to term 107
I20260501 08:11:11.583576 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.583766 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 107 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:11.584151 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 107 candidate_status { last_received { term: 106 index: 96 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:11.584201 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 107 candidate_status { last_received { term: 106 index: 96 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:11.584311 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 107 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 107: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:11.585036 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 107 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 107.
I20260501 08:11:11.585335 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 107 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:11.585456 1531 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 107 FOLLOWER]: Leader election won for term 107
I20260501 08:11:11.585551 1531 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 107 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:11.585632 1531 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 106.96, Last appended by leader: 96, Current term: 107, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.586130 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 106 to 107, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 107 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:11.656358 1552 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 107 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:11.656484 1552 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 107 FOLLOWER]: Advancing to term 108
I20260501 08:11:11.658502 1552 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.658810 1552 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 108 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:11.659190 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 108 candidate_status { last_received { term: 106 index: 96 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:11.659212 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 108 candidate_status { last_received { term: 106 index: 96 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:11.659436 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 107 FOLLOWER]: Advancing to term 108
I20260501 08:11:11.660599 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 108 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 108.
I20260501 08:11:11.660882 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 108 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:11.661103 1552 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 108 FOLLOWER]: Leader election won for term 108
I20260501 08:11:11.661273 1552 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 108 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:11.661360 1552 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 106.96, Last appended by leader: 96, Current term: 108, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.661962 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 107 to 108, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 108 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:11.700634 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 108 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 107. Current term is 108. Ops: []
I20260501 08:11:11.700990 1531 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20260501 08:11:11.701139 1531 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 108.97 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 96
I20260501 08:11:11.701174 1551 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 107 LEADER]: Stepping down as leader of term 107
I20260501 08:11:11.701238 1551 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 107 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:11.701330 1551 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 107.97, Last appended by leader: 97, Current term: 107, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:11.701417 1551 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 107 FOLLOWER]: Advancing to term 108
I20260501 08:11:11.772599 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 108 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 107 index: 97. Preceding OpId from leader: term: 108 index: 97. (term mismatch)
I20260501 08:11:11.772687 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 96
I20260501 08:11:11.772702 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 107.97
I20260501 08:11:11.772717 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 108 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:11.773087 1552 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20260501 08:11:11.774514 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 108 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 106 index: 96. Preceding OpId from leader: term: 108 index: 97. (index mismatch)
I20260501 08:11:11.774838 1552 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20260501 08:11:12.058808 1549 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 108 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:12.058935 1549 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 108 FOLLOWER]: Advancing to term 109
I20260501 08:11:12.059648 1549 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.059876 1549 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 109 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:12.060171 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 109 candidate_status { last_received { term: 108 index: 97 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:12.060171 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 109 candidate_status { last_received { term: 108 index: 97 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:12.060446 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 109 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:12.060534 1549 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 109 FOLLOWER]: Leader election lost for term 109. Reason: could not achieve majority
I20260501 08:11:12.079523 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 109 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 108. Current term is 109. Ops: []
I20260501 08:11:12.080062 1556 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 108.97, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20260501 08:11:12.080395 1556 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 108 LEADER]: Stepping down as leader of term 108
I20260501 08:11:12.080456 1556 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 108 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:12.080549 1556 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 108.97, Last appended by leader: 97, Current term: 108, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.080686 1556 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 108 FOLLOWER]: Advancing to term 109
I20260501 08:11:12.085259 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:12.122757 1551 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 108 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:12.122880 1551 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 108 FOLLOWER]: Advancing to term 109
I20260501 08:11:12.123588 1551 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.123776 1551 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 109 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:12.124004 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 109 candidate_status { last_received { term: 108 index: 97 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:12.124083 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 109 candidate_status { last_received { term: 108 index: 97 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:12.124094 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 109 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 109: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:12.124898 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 109.
I20260501 08:11:12.125141 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 109 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:12.125285 1551 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 109 FOLLOWER]: Leader election won for term 109
I20260501 08:11:12.125373 1551 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 109 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:12.125475 1551 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 108.97, Last appended by leader: 97, Current term: 109, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.126012 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 108 to 109, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 109 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:12.164018 1549 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 109 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:12.164179 1549 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 109 FOLLOWER]: Advancing to term 110
I20260501 08:11:12.165112 1549 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.165351 1549 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 110 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:12.165612 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 110 candidate_status { last_received { term: 108 index: 97 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:12.165712 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 110 candidate_status { last_received { term: 108 index: 97 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:12.165796 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 109 FOLLOWER]: Advancing to term 110
I20260501 08:11:12.166405 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 110 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 110.
I20260501 08:11:12.166587 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 110 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:12.166684 1549 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 110 FOLLOWER]: Leader election won for term 110
I20260501 08:11:12.166805 1549 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 110 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:12.166875 1549 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 108.97, Last appended by leader: 97, Current term: 110, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.167491 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 109 to 110, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 110 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:12.206593 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 110 LEADER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 109. Current term is 110. Ops: []
I20260501 08:11:12.206959 1551 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20260501 08:11:12.207106 1551 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 110.98 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 97
I20260501 08:11:12.207152 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 109 LEADER]: Stepping down as leader of term 109
I20260501 08:11:12.207206 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 109 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:12.207283 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 109.98, Last appended by leader: 98, Current term: 109, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.207381 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 109 FOLLOWER]: Advancing to term 110
I20260501 08:11:12.259672 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 110 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 108 index: 97. Preceding OpId from leader: term: 110 index: 98. (index mismatch)
I20260501 08:11:12.259994 1549 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20260501 08:11:12.261240 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 110 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 109 index: 98. Preceding OpId from leader: term: 110 index: 98. (term mismatch)
I20260501 08:11:12.261298 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 97
I20260501 08:11:12.261317 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 109.98
I20260501 08:11:12.261340 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 110 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:12.261483 1549 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20260501 08:11:12.361711 1556 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 110 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:12.361830 1556 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 110 FOLLOWER]: Advancing to term 111
I20260501 08:11:12.362522 1556 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.362725 1556 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 111 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:12.362967 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 111 candidate_status { last_received { term: 110 index: 98 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:12.362967 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 111 candidate_status { last_received { term: 110 index: 98 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:12.363097 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 110 FOLLOWER]: Advancing to term 111
I20260501 08:11:12.363758 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 111 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 111.
I20260501 08:11:12.364091 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 111 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:12.364198 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 111 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 110. Current term is 111. Ops: []
I20260501 08:11:12.364316 1556 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 111 FOLLOWER]: Leader election won for term 111
I20260501 08:11:12.364380 1556 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 111 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:12.364471 1549 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 110.98, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20260501 08:11:12.364477 1556 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 110.98, Last appended by leader: 98, Current term: 111, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.364601 1549 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 110 LEADER]: Stepping down as leader of term 110
I20260501 08:11:12.364637 1549 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 110 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:12.364774 1549 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 110.98, Last appended by leader: 98, Current term: 110, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.364912 1549 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 110 FOLLOWER]: Advancing to term 111
I20260501 08:11:12.365024 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 110 to 111, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 111 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:12.448807 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 111 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 110 index: 98. Preceding OpId from leader: term: 111 index: 99. (index mismatch)
I20260501 08:11:12.449170 1556 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20260501 08:11:12.450591 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 111 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 110 index: 98. Preceding OpId from leader: term: 111 index: 99. (index mismatch)
I20260501 08:11:12.450834 1558 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20260501 08:11:12.721105 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 111 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:12.721271 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 111 FOLLOWER]: Advancing to term 112
I20260501 08:11:12.721973 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.722184 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 112 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:12.722414 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 112 candidate_status { last_received { term: 111 index: 99 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:12.722416 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 112 candidate_status { last_received { term: 111 index: 99 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:12.722695 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 112 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:12.722843 1531 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 112 FOLLOWER]: Leader election lost for term 112. Reason: could not achieve majority
I20260501 08:11:12.737862 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 112 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 111. Current term is 112. Ops: []
I20260501 08:11:12.738224 1556 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 111.99, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20260501 08:11:12.738366 1555 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 111 LEADER]: Stepping down as leader of term 111
I20260501 08:11:12.738409 1555 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 111 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:12.738492 1555 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 111.99, Last appended by leader: 99, Current term: 111, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.738579 1555 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 111 FOLLOWER]: Advancing to term 112
I20260501 08:11:12.828315 1557 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 111 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:12.828431 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 111 FOLLOWER]: Advancing to term 112
I20260501 08:11:12.829210 1557 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.829404 1557 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 112 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:12.829625 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 112 candidate_status { last_received { term: 111 index: 99 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:12.829632 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 112 candidate_status { last_received { term: 111 index: 99 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:12.829789 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 112 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 112: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:12.830354 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 112 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 112.
I20260501 08:11:12.830565 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 112 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:12.830667 1557 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 112 FOLLOWER]: Leader election won for term 112
I20260501 08:11:12.830729 1557 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 112 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:12.830785 1557 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 111.99, Last appended by leader: 99, Current term: 112, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.831349 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 111 to 112, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 112 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:12.856390 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 112 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:12.856539 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 112 FOLLOWER]: Advancing to term 113
I20260501 08:11:12.857369 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.857589 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 113 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:12.857800 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 113 candidate_status { last_received { term: 111 index: 99 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:12.857824 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 113 candidate_status { last_received { term: 111 index: 99 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:12.857889 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 112 FOLLOWER]: Advancing to term 113
I20260501 08:11:12.858616 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 113.
I20260501 08:11:12.858850 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 113 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:12.859019 1531 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 113 FOLLOWER]: Leader election won for term 113
I20260501 08:11:12.859172 1531 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 113 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:12.859253 1531 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 111.99, Last appended by leader: 99, Current term: 113, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.859781 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 112 to 113, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 113 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:12.912025 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 113 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 112. Current term is 113. Ops: []
I20260501 08:11:12.912352 1557 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20260501 08:11:12.912501 1557 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 112 LEADER]: Stepping down as leader of term 112
I20260501 08:11:12.912536 1557 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 112 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:12.912608 1557 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 112.100, Last appended by leader: 100, Current term: 112, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:12.912683 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 112 FOLLOWER]: Advancing to term 113
I20260501 08:11:12.950487 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 113 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 112 index: 100. Preceding OpId from leader: term: 113 index: 100. (term mismatch)
I20260501 08:11:12.950575 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 99
I20260501 08:11:12.950590 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 112.100
I20260501 08:11:12.950605 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 113 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:12.950986 1531 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20260501 08:11:12.952366 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 113 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 111 index: 99. Preceding OpId from leader: term: 113 index: 100. (index mismatch)
I20260501 08:11:12.952553 1559 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20260501 08:11:13.053876 1555 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 113 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:13.054047 1555 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 113 FOLLOWER]: Advancing to term 114
I20260501 08:11:13.054742 1555 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.054963 1555 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 114 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:13.055186 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 114 candidate_status { last_received { term: 113 index: 100 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:13.055233 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 114 candidate_status { last_received { term: 113 index: 100 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:13.055469 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 114 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:13.055572 1555 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 114 FOLLOWER]: Leader election lost for term 114. Reason: could not achieve majority
I20260501 08:11:13.076953 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 114 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 113. Current term is 114. Ops: []
I20260501 08:11:13.077291 1531 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 113.100, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20260501 08:11:13.077416 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 113 LEADER]: Stepping down as leader of term 113
I20260501 08:11:13.077440 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 113 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:13.077497 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 113.100, Last appended by leader: 100, Current term: 113, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.077581 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 113 FOLLOWER]: Advancing to term 114
I20260501 08:11:13.085588 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:13.143378 1557 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 113 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:13.143491 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 113 FOLLOWER]: Advancing to term 114
I20260501 08:11:13.144371 1557 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.144553 1557 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 114 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:13.144804 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 114 candidate_status { last_received { term: 113 index: 100 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:13.144824 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 114 candidate_status { last_received { term: 113 index: 100 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:13.144933 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 114 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 114: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:13.145475 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 114 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 114.
I20260501 08:11:13.145689 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 114 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:13.145815 1557 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 114 FOLLOWER]: Leader election won for term 114
I20260501 08:11:13.145901 1557 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 114 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:13.146092 1557 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 113.100, Last appended by leader: 100, Current term: 114, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.146664 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 113 to 114, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 114 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:13.161283 1555 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 114 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:13.161391 1555 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 114 FOLLOWER]: Advancing to term 115
I20260501 08:11:13.162083 1555 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.162286 1555 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 115 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:13.162499 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 115 candidate_status { last_received { term: 113 index: 100 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:13.162508 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 115 candidate_status { last_received { term: 113 index: 100 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:13.162676 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 114 FOLLOWER]: Advancing to term 115
I20260501 08:11:13.163339 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 115.
I20260501 08:11:13.163589 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 115 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:13.163712 1555 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 115 FOLLOWER]: Leader election won for term 115
I20260501 08:11:13.163780 1555 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 115 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:13.163836 1555 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 113.100, Last appended by leader: 100, Current term: 115, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.164341 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 114 to 115, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 115 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:13.254186 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 115 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 114. Current term is 115. Ops: []
I20260501 08:11:13.254627 1557 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20260501 08:11:13.254738 1557 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 115.101 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 100
I20260501 08:11:13.254763 1557 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 114 LEADER]: Stepping down as leader of term 114
I20260501 08:11:13.254777 1557 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 114 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:13.254837 1557 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 114.101, Last appended by leader: 101, Current term: 114, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.254889 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 114 FOLLOWER]: Advancing to term 115
I20260501 08:11:13.256021 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 115 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 113 index: 100. Preceding OpId from leader: term: 115 index: 101. (index mismatch)
I20260501 08:11:13.256434 1555 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20260501 08:11:13.257719 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 115 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 114 index: 101. Preceding OpId from leader: term: 115 index: 101. (term mismatch)
I20260501 08:11:13.257771 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 100
I20260501 08:11:13.257795 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 114.101
I20260501 08:11:13.257817 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 115 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:13.257962 1555 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20260501 08:11:13.539858 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 115 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:13.540163 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 115 FOLLOWER]: Advancing to term 116
I20260501 08:11:13.540838 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.541066 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 116 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:13.541276 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 116 candidate_status { last_received { term: 115 index: 101 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:13.541282 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 116 candidate_status { last_received { term: 115 index: 101 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:13.541535 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 116 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:13.541677 1531 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 116 FOLLOWER]: Leader election lost for term 116. Reason: could not achieve majority
I20260501 08:11:13.549962 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 116 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 115. Current term is 116. Ops: []
I20260501 08:11:13.550355 1561 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 115.101, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20260501 08:11:13.550470 1561 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 115 LEADER]: Stepping down as leader of term 115
I20260501 08:11:13.550488 1561 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 115 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:13.550576 1561 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 115.101, Last appended by leader: 101, Current term: 115, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.550660 1561 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 115 FOLLOWER]: Advancing to term 116
I20260501 08:11:13.620877 1549 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 115 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:13.621019 1549 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 115 FOLLOWER]: Advancing to term 116
I20260501 08:11:13.621757 1549 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.621950 1549 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 116 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:13.622265 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 116 candidate_status { last_received { term: 115 index: 101 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:13.622241 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 116 candidate_status { last_received { term: 115 index: 101 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:13.622349 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 116 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 116: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:13.623018 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 116 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 116.
I20260501 08:11:13.623262 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 116 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:13.623358 1549 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 116 FOLLOWER]: Leader election won for term 116
I20260501 08:11:13.623413 1549 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 116 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:13.623458 1549 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 115.101, Last appended by leader: 101, Current term: 116, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.623968 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 115 to 116, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 116 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:13.650156 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 116 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:13.650261 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 116 FOLLOWER]: Advancing to term 117
I20260501 08:11:13.650983 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.651204 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 117 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:13.651427 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 117 candidate_status { last_received { term: 115 index: 101 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:13.651491 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 116 FOLLOWER]: Advancing to term 117
I20260501 08:11:13.651520 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 117 candidate_status { last_received { term: 115 index: 101 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:13.652415 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 117.
I20260501 08:11:13.652684 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 117 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:13.652776 1531 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 117 FOLLOWER]: Leader election won for term 117
I20260501 08:11:13.652882 1531 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 117 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:13.652954 1531 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 115.101, Last appended by leader: 101, Current term: 117, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.653574 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 116 to 117, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 117 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:13.724772 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 117 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 116. Current term is 117. Ops: []
I20260501 08:11:13.725286 1549 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20260501 08:11:13.725477 1549 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 117.102 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 101
I20260501 08:11:13.725522 1557 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 116 LEADER]: Stepping down as leader of term 116
I20260501 08:11:13.725571 1557 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 116 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:13.725626 1557 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 116.102, Last appended by leader: 102, Current term: 116, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.725696 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 116 FOLLOWER]: Advancing to term 117
I20260501 08:11:13.733372 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 117 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 115 index: 101. Preceding OpId from leader: term: 117 index: 102. (index mismatch)
I20260501 08:11:13.733729 1531 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20260501 08:11:13.735050 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 117 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 116 index: 102. Preceding OpId from leader: term: 117 index: 102. (term mismatch)
I20260501 08:11:13.735107 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 101
I20260501 08:11:13.735131 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 116.102
I20260501 08:11:13.735148 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 117 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:13.735311 1563 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20260501 08:11:13.953990 1557 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 117 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:13.954108 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 117 FOLLOWER]: Advancing to term 118
I20260501 08:11:13.954876 1557 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.955106 1557 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 118 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:13.955363 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 118 candidate_status { last_received { term: 117 index: 102 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:13.955385 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 118 candidate_status { last_received { term: 117 index: 102 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:13.955607 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 118 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:13.955729 1557 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 118 FOLLOWER]: Leader election lost for term 118. Reason: could not achieve majority
I20260501 08:11:13.959882 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 118 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 117. Current term is 118. Ops: []
I20260501 08:11:13.960280 1562 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 117.102, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20260501 08:11:13.960394 1562 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 117 LEADER]: Stepping down as leader of term 117
I20260501 08:11:13.960414 1562 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 117 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:13.960491 1562 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 117.102, Last appended by leader: 102, Current term: 117, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:13.960598 1562 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 117 FOLLOWER]: Advancing to term 118
I20260501 08:11:14.024989 1561 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 117 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:14.025106 1561 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 117 FOLLOWER]: Advancing to term 118
I20260501 08:11:14.025808 1561 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.026026 1561 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 118 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:14.026247 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 118 candidate_status { last_received { term: 117 index: 102 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:14.026242 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 118 candidate_status { last_received { term: 117 index: 102 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:14.026418 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 118 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 118: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:14.027061 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 118 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 118.
I20260501 08:11:14.027264 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 118 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:14.027477 1561 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 118 FOLLOWER]: Leader election won for term 118
I20260501 08:11:14.027575 1561 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 118 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:14.027640 1561 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 117.102, Last appended by leader: 102, Current term: 118, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.028224 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 117 to 118, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 118 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:14.070247 1557 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 118 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:14.070374 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 118 FOLLOWER]: Advancing to term 119
I20260501 08:11:14.071159 1557 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.071373 1557 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 119 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:14.071655 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 119 candidate_status { last_received { term: 117 index: 102 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:14.071655 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 119 candidate_status { last_received { term: 117 index: 102 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:14.071770 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 118 FOLLOWER]: Advancing to term 119
I20260501 08:11:14.072539 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 119 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 119.
I20260501 08:11:14.072777 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 119 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:14.072940 1557 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 119 FOLLOWER]: Leader election won for term 119
I20260501 08:11:14.073115 1557 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 119 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:14.073261 1557 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 117.102, Last appended by leader: 102, Current term: 119, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.073855 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 118 to 119, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 119 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:14.085924 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:14.104772 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 119 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 118. Current term is 119. Ops: []
I20260501 08:11:14.105201 1561 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20260501 08:11:14.105361 1561 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 118 LEADER]: Stepping down as leader of term 118
I20260501 08:11:14.105392 1561 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 118 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:14.105471 1561 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 118.103, Last appended by leader: 103, Current term: 118, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.105551 1561 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 118 FOLLOWER]: Advancing to term 119
I20260501 08:11:14.166347 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 119 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 118 index: 103. Preceding OpId from leader: term: 119 index: 103. (term mismatch)
I20260501 08:11:14.166429 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 102
I20260501 08:11:14.166443 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 118.103
I20260501 08:11:14.166458 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 119 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:14.166747 1557 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20260501 08:11:14.168175 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 119 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 117 index: 102. Preceding OpId from leader: term: 119 index: 103. (index mismatch)
I20260501 08:11:14.168393 1549 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20260501 08:11:14.269840 1562 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 119 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:14.269992 1562 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 119 FOLLOWER]: Advancing to term 120
I20260501 08:11:14.270768 1562 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.271157 1562 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 120 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:14.271270 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 120 candidate_status { last_received { term: 119 index: 103 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:14.271322 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 120 candidate_status { last_received { term: 119 index: 103 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:14.271384 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 119 FOLLOWER]: Advancing to term 120
I20260501 08:11:14.271992 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 120 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 120.
I20260501 08:11:14.272285 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 120 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:14.272392 1562 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 120 FOLLOWER]: Leader election won for term 120
I20260501 08:11:14.272442 1562 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 120 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:14.272495 1562 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 119.103, Last appended by leader: 103, Current term: 120, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.272993 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 119 to 120, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 120 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:14.274899 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 120 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 119. Current term is 120. Ops: []
I20260501 08:11:14.275164 1549 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 119.103, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20260501 08:11:14.275269 1549 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 120.104 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 103
I20260501 08:11:14.275312 1557 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 119 LEADER]: Stepping down as leader of term 119
I20260501 08:11:14.275360 1557 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 119 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:14.275420 1557 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 119.103, Last appended by leader: 103, Current term: 119, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.275521 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 119 FOLLOWER]: Advancing to term 120
I20260501 08:11:14.372735 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 120 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 119 index: 103. Preceding OpId from leader: term: 120 index: 104. (index mismatch)
I20260501 08:11:14.373282 1562 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20260501 08:11:14.374383 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 120 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 119 index: 103. Preceding OpId from leader: term: 120 index: 104. (index mismatch)
I20260501 08:11:14.374619 1564 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20260501 08:11:14.565510 1557 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 120 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:14.565625 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 120 FOLLOWER]: Advancing to term 121
I20260501 08:11:14.565966 1561 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 120 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:14.566021 1561 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 120 FOLLOWER]: Advancing to term 121
I20260501 08:11:14.566540 1557 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.566728 1557 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 121 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:14.566805 1561 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.566933 1561 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 121 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:14.566965 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 121 candidate_status { last_received { term: 120 index: 104 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:14.567039 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 121 candidate_status { last_received { term: 120 index: 104 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:14.567078 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 121 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 121: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:14.567101 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 121 candidate_status { last_received { term: 120 index: 104 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:14.567168 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 121 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 121: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:14.567247 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 121 candidate_status { last_received { term: 120 index: 104 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:14.567298 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 121 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:14.567412 1557 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 121 FOLLOWER]: Leader election lost for term 121. Reason: could not achieve majority
I20260501 08:11:14.567421 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 121 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:14.567554 1561 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 121 FOLLOWER]: Leader election lost for term 121. Reason: could not achieve majority
I20260501 08:11:14.569998 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 121 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 120. Current term is 121. Ops: []
I20260501 08:11:14.570401 1531 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 120.104, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20260501 08:11:14.570513 1531 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 120 LEADER]: Stepping down as leader of term 120
I20260501 08:11:14.570530 1531 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 120 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:14.570631 1531 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 120.104, Last appended by leader: 104, Current term: 120, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.570734 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 120 FOLLOWER]: Advancing to term 121
I20260501 08:11:14.707178 1561 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 121 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:14.707336 1561 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 121 FOLLOWER]: Advancing to term 122
I20260501 08:11:14.708079 1561 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.708377 1561 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 122 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:14.708645 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 122 candidate_status { last_received { term: 120 index: 104 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:14.708717 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 121 FOLLOWER]: Advancing to term 122
I20260501 08:11:14.708640 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 122 candidate_status { last_received { term: 120 index: 104 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:14.708797 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 121 FOLLOWER]: Advancing to term 122
I20260501 08:11:14.709378 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 122 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 122.
I20260501 08:11:14.709587 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 122 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 122.
I20260501 08:11:14.709623 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 122 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters:
I20260501 08:11:14.709707 1561 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 122 FOLLOWER]: Leader election won for term 122
I20260501 08:11:14.709746 1561 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 122 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:14.709791 1561 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 120.104, Last appended by leader: 104, Current term: 122, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.710372 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 120 to 122, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 122 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:14.789839 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 122 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 120 index: 104. Preceding OpId from leader: term: 122 index: 105. (index mismatch)
I20260501 08:11:14.790141 1561 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20260501 08:11:14.791215 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 122 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 120 index: 104. Preceding OpId from leader: term: 122 index: 105. (index mismatch)
I20260501 08:11:14.791405 1565 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20260501 08:11:14.891508 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 122 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:14.891662 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 122 FOLLOWER]: Advancing to term 123
I20260501 08:11:14.892330 1557 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 122 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:14.892388 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 122 FOLLOWER]: Advancing to term 123
I20260501 08:11:14.892557 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.892771 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 123 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:14.892961 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 123 candidate_status { last_received { term: 122 index: 105 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:14.892959 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 123 candidate_status { last_received { term: 122 index: 105 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:14.893138 1557 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.893196 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 123 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 123: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:14.893252 1557 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 123 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:14.893399 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 123 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:14.893425 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 123 candidate_status { last_received { term: 122 index: 105 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:14.893446 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 123 candidate_status { last_received { term: 122 index: 105 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:14.893488 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 123 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 123: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:14.893518 1531 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 123 FOLLOWER]: Leader election lost for term 123. Reason: could not achieve majority
I20260501 08:11:14.893615 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 123 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:14.893718 1557 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 123 FOLLOWER]: Leader election lost for term 123. Reason: could not achieve majority
I20260501 08:11:14.895030 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 123 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 122. Current term is 123. Ops: []
I20260501 08:11:14.895326 1565 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 122.105, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20260501 08:11:14.895435 1565 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 122 LEADER]: Stepping down as leader of term 122
I20260501 08:11:14.895471 1565 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 122 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:14.895540 1565 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 122.105, Last appended by leader: 105, Current term: 122, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:14.895617 1565 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 122 FOLLOWER]: Advancing to term 123
I20260501 08:11:15.021174 1557 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 123 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:15.021302 1557 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 123 FOLLOWER]: Advancing to term 124
I20260501 08:11:15.021993 1557 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.022234 1557 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 124 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:15.022459 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 124 candidate_status { last_received { term: 122 index: 105 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:15.022536 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 123 FOLLOWER]: Advancing to term 124
I20260501 08:11:15.022493 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 124 candidate_status { last_received { term: 122 index: 105 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:15.022575 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 123 FOLLOWER]: Advancing to term 124
I20260501 08:11:15.023149 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 124 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 124.
I20260501 08:11:15.023154 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 124 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 124.
I20260501 08:11:15.023417 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 124 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters:
I20260501 08:11:15.023502 1557 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 124 FOLLOWER]: Leader election won for term 124
I20260501 08:11:15.023615 1557 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 124 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:15.023660 1557 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 122.105, Last appended by leader: 105, Current term: 124, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.024237 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 122 to 124, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 124 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:15.086251 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:15.117920 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 124 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 122 index: 105. Preceding OpId from leader: term: 124 index: 106. (index mismatch)
I20260501 08:11:15.118009 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 124 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 122 index: 105. Preceding OpId from leader: term: 124 index: 106. (index mismatch)
I20260501 08:11:15.118251 1557 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20260501 08:11:15.118412 1566 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20260501 08:11:15.219776 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 124 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:15.219894 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 124 FOLLOWER]: Advancing to term 125
I20260501 08:11:15.220679 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.220904 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 125 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:15.221215 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 125 candidate_status { last_received { term: 124 index: 106 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:15.221262 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 125 candidate_status { last_received { term: 124 index: 106 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:15.221524 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 125 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:15.221655 1531 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 125 FOLLOWER]: Leader election lost for term 125. Reason: could not achieve majority
I20260501 08:11:15.222335 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 125 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 124. Current term is 125. Ops: []
I20260501 08:11:15.222613 1567 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 124.106, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20260501 08:11:15.222765 1566 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 124 LEADER]: Stepping down as leader of term 124
I20260501 08:11:15.222815 1566 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 124 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:15.222893 1566 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 124.106, Last appended by leader: 106, Current term: 124, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.223002 1566 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 124 FOLLOWER]: Advancing to term 125
I20260501 08:11:15.311802 1565 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 124 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:15.311923 1565 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 124 FOLLOWER]: Advancing to term 125
I20260501 08:11:15.312711 1565 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.313028 1565 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 125 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:15.313163 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 125 candidate_status { last_received { term: 124 index: 106 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:15.313163 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 125 candidate_status { last_received { term: 124 index: 106 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:15.313303 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 125 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 125: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:15.313959 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 125 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 125.
I20260501 08:11:15.314183 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 125 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:15.314304 1565 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 125 FOLLOWER]: Leader election won for term 125
I20260501 08:11:15.314399 1565 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 125 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:15.314457 1565 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 124.106, Last appended by leader: 106, Current term: 125, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.314949 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 124 to 125, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 125 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:15.363161 1531 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 125 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:15.363274 1531 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 125 FOLLOWER]: Advancing to term 126
I20260501 08:11:15.364020 1531 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.364272 1531 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 126 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:15.364517 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 126 candidate_status { last_received { term: 124 index: 106 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:15.364517 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 126 candidate_status { last_received { term: 124 index: 106 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:15.364672 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 125 FOLLOWER]: Advancing to term 126
I20260501 08:11:15.365288 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 126 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 126.
I20260501 08:11:15.365573 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 126 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:15.365706 1531 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 126 FOLLOWER]: Leader election won for term 126
I20260501 08:11:15.365859 1531 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 126 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:15.365942 1531 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 124.106, Last appended by leader: 106, Current term: 126, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.366423 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 125 to 126, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 126 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:15.419528 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 126 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 125. Current term is 126. Ops: []
I20260501 08:11:15.419875 1565 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20260501 08:11:15.420018 1565 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 125 LEADER]: Stepping down as leader of term 125
I20260501 08:11:15.420075 1565 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 125 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:15.420176 1565 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 125.107, Last appended by leader: 107, Current term: 125, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.420262 1565 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 125 FOLLOWER]: Advancing to term 126
I20260501 08:11:15.443861 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 126 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 124 index: 106. Preceding OpId from leader: term: 126 index: 107. (index mismatch)
I20260501 08:11:15.444247 1531 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20260501 08:11:15.445612 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 126 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 125 index: 107. Preceding OpId from leader: term: 126 index: 107. (term mismatch)
I20260501 08:11:15.445672 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 106
I20260501 08:11:15.445698 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 125.107
I20260501 08:11:15.445719 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 126 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:15.445910 1531 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20260501 08:11:15.624639 1566 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 126 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:15.624761 1566 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 126 FOLLOWER]: Advancing to term 127
I20260501 08:11:15.625571 1566 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.625764 1566 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 127 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:15.626008 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 127 candidate_status { last_received { term: 126 index: 107 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:15.626035 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 127 candidate_status { last_received { term: 126 index: 107 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:15.626276 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 127 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:15.626374 1566 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 127 FOLLOWER]: Leader election lost for term 127. Reason: could not achieve majority
I20260501 08:11:15.633370 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 127 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 126. Current term is 127. Ops: []
I20260501 08:11:15.634392 1569 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 126.107, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20260501 08:11:15.635026 1568 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 126 LEADER]: Stepping down as leader of term 126
I20260501 08:11:15.635149 1568 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 126 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:15.635246 1568 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 126.107, Last appended by leader: 107, Current term: 126, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.635417 1568 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 126 FOLLOWER]: Advancing to term 127
I20260501 08:11:15.645670 1556 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 126 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:15.645783 1556 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 126 FOLLOWER]: Advancing to term 127
I20260501 08:11:15.646518 1556 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.646713 1556 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 127 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:15.646924 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 127 candidate_status { last_received { term: 126 index: 107 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:15.646930 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 127 candidate_status { last_received { term: 126 index: 107 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:15.647075 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 127 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 127: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:15.647600 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 127 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 127.
I20260501 08:11:15.647811 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 127 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters:
I20260501 08:11:15.647907 1556 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 127 FOLLOWER]: Leader election won for term 127
I20260501 08:11:15.647964 1556 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 127 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:15.648005 1556 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 126.107, Last appended by leader: 107, Current term: 127, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.648532 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 126 to 127, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 127 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:15.743228 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 127 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 126 index: 107. Preceding OpId from leader: term: 127 index: 108. (index mismatch)
I20260501 08:11:15.743558 1556 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20260501 08:11:15.744567 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 127 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 126 index: 107. Preceding OpId from leader: term: 127 index: 108. (index mismatch)
I20260501 08:11:15.744817 1570 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20260501 08:11:15.845625 1566 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 127 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:15.845767 1566 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 127 FOLLOWER]: Advancing to term 128
I20260501 08:11:15.846701 1566 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.846849 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 128 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 127. Current term is 128. Ops: []
I20260501 08:11:15.847008 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 128 candidate_status { last_received { term: 127 index: 108 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:15.847013 1566 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 128 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:15.847040 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 128 candidate_status { last_received { term: 127 index: 108 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:15.847132 1565 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 127.108, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20260501 08:11:15.847222 1565 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 127 LEADER]: Stepping down as leader of term 127
I20260501 08:11:15.847281 1565 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 127 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:15.847335 1565 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 127.108, Last appended by leader: 108, Current term: 127, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.847611 1565 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 127 FOLLOWER]: Advancing to term 128
I20260501 08:11:15.847782 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 128 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:15.847873 1566 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 128 FOLLOWER]: Leader election lost for term 128. Reason: could not achieve majority
I20260501 08:11:15.937703 1568 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 127 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:15.937847 1568 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 127 FOLLOWER]: Advancing to term 128
I20260501 08:11:15.938570 1568 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.938753 1568 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 128 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:15.938997 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 128 candidate_status { last_received { term: 127 index: 108 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:15.939038 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 128 candidate_status { last_received { term: 127 index: 108 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:15.939189 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 128 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 128: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:15.939690 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 128 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 128.
I20260501 08:11:15.939972 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 128 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:15.940114 1568 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 128 FOLLOWER]: Leader election won for term 128
I20260501 08:11:15.940235 1568 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 128 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:15.940301 1568 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 127.108, Last appended by leader: 108, Current term: 128, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.940850 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 127 to 128, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 128 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:15.954367 1566 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 128 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:15.954478 1566 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 128 FOLLOWER]: Advancing to term 129
I20260501 08:11:15.955214 1566 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.955415 1566 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 129 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:15.955631 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 129 candidate_status { last_received { term: 127 index: 108 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:15.955641 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 129 candidate_status { last_received { term: 127 index: 108 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:15.955736 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 128 FOLLOWER]: Advancing to term 129
I20260501 08:11:15.956615 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 129.
I20260501 08:11:15.956856 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 129 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:15.956959 1566 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 129 FOLLOWER]: Leader election won for term 129
I20260501 08:11:15.957085 1566 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 129 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:15.957132 1566 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 127.108, Last appended by leader: 108, Current term: 129, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:15.957681 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 128 to 129, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 129 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:16.038363 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 129 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 128. Current term is 129. Ops: []
I20260501 08:11:16.038725 1568 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20260501 08:11:16.038866 1568 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 128 LEADER]: Stepping down as leader of term 128
I20260501 08:11:16.038920 1568 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 128 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:16.038993 1568 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 128.109, Last appended by leader: 109, Current term: 128, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.039108 1568 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 128 FOLLOWER]: Advancing to term 129
I20260501 08:11:16.043354 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 129 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 128 index: 109. Preceding OpId from leader: term: 129 index: 109. (term mismatch)
I20260501 08:11:16.043421 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 108
I20260501 08:11:16.043434 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 128.109
I20260501 08:11:16.043449 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 129 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:16.043689 1566 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20260501 08:11:16.045212 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 129 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 127 index: 108. Preceding OpId from leader: term: 129 index: 109. (index mismatch)
I20260501 08:11:16.045475 1566 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20260501 08:11:16.086534 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:16.265194 1568 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 129 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:16.265300 1568 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 129 FOLLOWER]: Advancing to term 130
I20260501 08:11:16.265976 1568 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.266203 1568 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 130 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:16.266410 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 130 candidate_status { last_received { term: 129 index: 109 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:16.266451 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 130 candidate_status { last_received { term: 129 index: 109 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:16.266772 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 130 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:16.267069 1568 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 130 FOLLOWER]: Leader election lost for term 130. Reason: could not achieve majority
I20260501 08:11:16.276405 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 130 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 129. Current term is 130. Ops: []
I20260501 08:11:16.276764 1572 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 129.109, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20260501 08:11:16.276917 1572 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 129 LEADER]: Stepping down as leader of term 129
I20260501 08:11:16.276953 1572 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 129 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:16.277037 1572 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 129.109, Last appended by leader: 109, Current term: 129, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.277143 1572 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 129 FOLLOWER]: Advancing to term 130
I20260501 08:11:16.302928 1565 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 129 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:16.303073 1565 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 129 FOLLOWER]: Advancing to term 130
I20260501 08:11:16.303809 1565 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.304008 1565 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 130 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:16.304312 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 130 candidate_status { last_received { term: 129 index: 109 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:16.304319 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 130 candidate_status { last_received { term: 129 index: 109 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:16.304450 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 130 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 130: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:16.304991 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 130 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 130.
I20260501 08:11:16.305204 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 130 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:16.305361 1565 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 130 FOLLOWER]: Leader election won for term 130
I20260501 08:11:16.305442 1565 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 130 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:16.305528 1565 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 129.109, Last appended by leader: 109, Current term: 130, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.306223 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 129 to 130, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 130 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:16.372188 1568 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 130 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:16.372300 1568 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 130 FOLLOWER]: Advancing to term 131
I20260501 08:11:16.372983 1568 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.373219 1568 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 131 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:16.373538 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 131 candidate_status { last_received { term: 129 index: 109 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:16.373682 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 131 candidate_status { last_received { term: 129 index: 109 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:16.373811 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 130 FOLLOWER]: Advancing to term 131
I20260501 08:11:16.374526 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 131 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 131.
I20260501 08:11:16.374845 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 131 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:16.375111 1568 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 131 FOLLOWER]: Leader election won for term 131
I20260501 08:11:16.375226 1568 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 131 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:16.375288 1568 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 129.109, Last appended by leader: 109, Current term: 131, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.375865 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 130 to 131, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 131 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:16.384622 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 131 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 130. Current term is 131. Ops: []
I20260501 08:11:16.384912 1565 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20260501 08:11:16.385037 1565 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 130 LEADER]: Stepping down as leader of term 130
I20260501 08:11:16.385056 1565 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 130 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:16.385128 1565 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 130.110, Last appended by leader: 110, Current term: 130, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.385206 1565 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 130 FOLLOWER]: Advancing to term 131
I20260501 08:11:16.460484 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 131 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 130 index: 110. Preceding OpId from leader: term: 131 index: 110. (term mismatch)
I20260501 08:11:16.460570 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 109
I20260501 08:11:16.460584 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 130.110
I20260501 08:11:16.460599 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 131 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:16.460865 1568 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20260501 08:11:16.461993 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 131 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 129 index: 109. Preceding OpId from leader: term: 131 index: 110. (index mismatch)
I20260501 08:11:16.462278 1569 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20260501 08:11:16.563446 1572 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 131 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:16.563540 1572 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 131 FOLLOWER]: Advancing to term 132
I20260501 08:11:16.564337 1572 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.564500 1572 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 132 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:16.564697 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 132 candidate_status { last_received { term: 131 index: 110 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:16.564720 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 132 candidate_status { last_received { term: 131 index: 110 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:16.564930 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 132 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:16.565029 1572 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 132 FOLLOWER]: Leader election lost for term 132. Reason: could not achieve majority
I20260501 08:11:16.576980 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 132 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 131. Current term is 132. Ops: []
I20260501 08:11:16.577313 1568 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 131.110, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20260501 08:11:16.577431 1568 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 131 LEADER]: Stepping down as leader of term 131
I20260501 08:11:16.577461 1568 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 131 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:16.577539 1568 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 131.110, Last appended by leader: 110, Current term: 131, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.577616 1568 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 131 FOLLOWER]: Advancing to term 132
I20260501 08:11:16.662926 1565 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 131 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:16.663045 1565 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 131 FOLLOWER]: Advancing to term 132
I20260501 08:11:16.664863 1565 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.665134 1565 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 132 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:16.665326 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 132 candidate_status { last_received { term: 131 index: 110 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:16.665423 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 132 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 132: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:16.665345 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 132 candidate_status { last_received { term: 131 index: 110 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:16.666033 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 132 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 132.
I20260501 08:11:16.666249 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 132 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:16.666384 1565 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 132 FOLLOWER]: Leader election won for term 132
I20260501 08:11:16.666476 1565 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 132 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:16.666569 1565 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 131.110, Last appended by leader: 110, Current term: 132, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.667135 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 131 to 132, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 132 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:16.670742 1572 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 132 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:16.670851 1572 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 132 FOLLOWER]: Advancing to term 133
I20260501 08:11:16.671518 1572 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.671705 1572 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 133 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:16.671900 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 133 candidate_status { last_received { term: 131 index: 110 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:16.671908 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 133 candidate_status { last_received { term: 131 index: 110 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:16.672022 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 132 FOLLOWER]: Advancing to term 133
I20260501 08:11:16.672629 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 133 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 133.
I20260501 08:11:16.672829 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 133 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:16.672931 1572 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 133 FOLLOWER]: Leader election won for term 133
I20260501 08:11:16.672984 1572 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 133 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:16.673033 1572 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 131.110, Last appended by leader: 110, Current term: 133, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.673607 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 132 to 133, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 133 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:16.753520 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 133 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 132. Current term is 133. Ops: []
I20260501 08:11:16.753957 1565 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20260501 08:11:16.754115 1565 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 132 LEADER]: Stepping down as leader of term 132
I20260501 08:11:16.754153 1565 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 132 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:16.754233 1565 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 132.111, Last appended by leader: 111, Current term: 132, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.754313 1565 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 132 FOLLOWER]: Advancing to term 133
I20260501 08:11:16.770064 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 133 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 131 index: 110. Preceding OpId from leader: term: 133 index: 111. (index mismatch)
I20260501 08:11:16.770453 1572 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20260501 08:11:16.771571 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 133 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 132 index: 111. Preceding OpId from leader: term: 133 index: 111. (term mismatch)
I20260501 08:11:16.771631 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 110
I20260501 08:11:16.771649 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 132.111
I20260501 08:11:16.771669 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 133 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:16.771824 1573 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20260501 08:11:16.872370 1568 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 133 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:16.872491 1568 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 133 FOLLOWER]: Advancing to term 134
I20260501 08:11:16.873216 1568 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.873441 1568 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 134 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:16.873667 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 134 candidate_status { last_received { term: 133 index: 111 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:16.873732 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 134 candidate_status { last_received { term: 133 index: 111 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:16.874061 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 134 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:16.874168 1568 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 134 FOLLOWER]: Leader election lost for term 134. Reason: could not achieve majority
I20260501 08:11:16.880647 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 134 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 133. Current term is 134. Ops: []
I20260501 08:11:16.881068 1572 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 133.111, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20260501 08:11:16.881246 1573 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 133 LEADER]: Stepping down as leader of term 133
I20260501 08:11:16.881286 1573 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 133 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:16.881366 1573 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 133.111, Last appended by leader: 111, Current term: 133, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.881449 1573 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 133 FOLLOWER]: Advancing to term 134
I20260501 08:11:16.980002 1565 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 133 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:16.980170 1565 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 133 FOLLOWER]: Advancing to term 134
I20260501 08:11:16.980888 1565 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.981101 1565 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 134 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:16.981330 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 134 candidate_status { last_received { term: 133 index: 111 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:16.981360 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 134 candidate_status { last_received { term: 133 index: 111 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:16.981456 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 134 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 134: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:16.982043 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 134 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 134.
I20260501 08:11:16.982266 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 134 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:16.982378 1565 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 134 FOLLOWER]: Leader election won for term 134
I20260501 08:11:16.982474 1565 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 134 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:16.982532 1565 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 133.111, Last appended by leader: 111, Current term: 134, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:16.983048 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 133 to 134, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 134 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:17.019380 1568 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 134 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:17.019481 1568 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 134 FOLLOWER]: Advancing to term 135
I20260501 08:11:17.020246 1568 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.020426 1568 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 135 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:17.020681 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 135 candidate_status { last_received { term: 133 index: 111 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:17.020681 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 135 candidate_status { last_received { term: 133 index: 111 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:17.020800 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 134 FOLLOWER]: Advancing to term 135
I20260501 08:11:17.021464 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 135 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 135.
I20260501 08:11:17.021682 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 135 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:17.021781 1568 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 135 FOLLOWER]: Leader election won for term 135
I20260501 08:11:17.021862 1568 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 135 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:17.021914 1568 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 133.111, Last appended by leader: 111, Current term: 135, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.022364 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 134 to 135, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 135 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:17.086862 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:17.097833 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 135 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 134. Current term is 135. Ops: []
I20260501 08:11:17.098315 1565 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20260501 08:11:17.098526 1565 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 135.112 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 111
I20260501 08:11:17.098603 1574 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 134 LEADER]: Stepping down as leader of term 134
I20260501 08:11:17.098662 1574 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 134 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:17.098722 1574 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 134.112, Last appended by leader: 112, Current term: 134, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.098811 1574 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 134 FOLLOWER]: Advancing to term 135
I20260501 08:11:17.098838 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 135 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 134. Current term is 135. Ops: []
I20260501 08:11:17.099524 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 135 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 133 index: 111. Preceding OpId from leader: term: 135 index: 112. (index mismatch)
I20260501 08:11:17.099797 1568 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20260501 08:11:17.101004 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 135 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 134 index: 112. Preceding OpId from leader: term: 135 index: 112. (term mismatch)
I20260501 08:11:17.101060 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 111
I20260501 08:11:17.101080 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 134.112
I20260501 08:11:17.101111 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 135 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:17.101289 1568 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20260501 08:11:17.284763 1573 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 135 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:17.284885 1573 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 135 FOLLOWER]: Advancing to term 136
I20260501 08:11:17.285738 1573 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.285959 1573 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 136 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:17.286264 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 136 candidate_status { last_received { term: 135 index: 112 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:17.286257 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 136 candidate_status { last_received { term: 135 index: 112 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:17.286628 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 136 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:17.286779 1573 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 136 FOLLOWER]: Leader election lost for term 136. Reason: could not achieve majority
I20260501 08:11:17.301719 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 136 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 135. Current term is 136. Ops: []
I20260501 08:11:17.302076 1569 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 135.112, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20260501 08:11:17.302227 1568 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 135 LEADER]: Stepping down as leader of term 135
I20260501 08:11:17.302275 1568 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 135 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:17.302345 1568 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 135.112, Last appended by leader: 112, Current term: 135, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.302423 1568 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 135 FOLLOWER]: Advancing to term 136
I20260501 08:11:17.375869 1570 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 135 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:17.376015 1570 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 135 FOLLOWER]: Advancing to term 136
I20260501 08:11:17.376885 1570 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.377079 1570 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 136 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:17.377328 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 136 candidate_status { last_received { term: 135 index: 112 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:17.377328 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 136 candidate_status { last_received { term: 135 index: 112 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:17.377457 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 136 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 136: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:17.378118 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 136.
I20260501 08:11:17.378377 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 136 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:17.378533 1570 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 136 FOLLOWER]: Leader election won for term 136
I20260501 08:11:17.378638 1570 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 136 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:17.378701 1570 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 135.112, Last appended by leader: 112, Current term: 136, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.379385 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 135 to 136, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 136 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:17.413348 1573 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 136 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:17.413468 1573 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 136 FOLLOWER]: Advancing to term 137
I20260501 08:11:17.414181 1573 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.414405 1573 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 137 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:17.414616 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 137 candidate_status { last_received { term: 135 index: 112 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:17.414628 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 137 candidate_status { last_received { term: 135 index: 112 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:17.414739 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 136 FOLLOWER]: Advancing to term 137
I20260501 08:11:17.415342 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 137 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 137.
I20260501 08:11:17.415575 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 137 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:17.415678 1573 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 137 FOLLOWER]: Leader election won for term 137
I20260501 08:11:17.415787 1573 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 137 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:17.415856 1573 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 135.112, Last appended by leader: 112, Current term: 137, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.416419 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 136 to 137, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 137 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:17.457528 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 137 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 136. Current term is 137. Ops: []
I20260501 08:11:17.457957 1570 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20260501 08:11:17.458130 1574 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 136 LEADER]: Stepping down as leader of term 136
I20260501 08:11:17.458184 1574 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 136 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:17.458268 1574 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 136.113, Last appended by leader: 113, Current term: 136, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.458365 1574 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 136 FOLLOWER]: Advancing to term 137
I20260501 08:11:17.512077 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 137 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 136 index: 113. Preceding OpId from leader: term: 137 index: 113. (term mismatch)
I20260501 08:11:17.512202 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 112
I20260501 08:11:17.512253 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 136.113
I20260501 08:11:17.512276 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 137 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:17.512579 1573 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20260501 08:11:17.513893 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 137 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 135 index: 112. Preceding OpId from leader: term: 137 index: 113. (index mismatch)
I20260501 08:11:17.514173 1577 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20260501 08:11:17.615345 1568 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 137 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:17.615484 1568 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 137 FOLLOWER]: Advancing to term 138
I20260501 08:11:17.616263 1568 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.616482 1568 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 138 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:17.616714 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 138 candidate_status { last_received { term: 137 index: 113 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:17.616720 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 138 candidate_status { last_received { term: 137 index: 113 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:17.616807 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 137 FOLLOWER]: Advancing to term 138
I20260501 08:11:17.617369 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 138 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 138.
I20260501 08:11:17.617547 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 138 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:17.617658 1568 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 138 FOLLOWER]: Leader election won for term 138
I20260501 08:11:17.617729 1568 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 138 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:17.617784 1568 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 137.113, Last appended by leader: 113, Current term: 138, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.618402 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 137 to 138, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 138 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:17.637542 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 138 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 137. Current term is 138. Ops: []
I20260501 08:11:17.637542 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 138 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 137. Current term is 138. Ops: []
I20260501 08:11:17.637881 1576 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 137.113, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20260501 08:11:17.638021 1576 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 138.114 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 113
I20260501 08:11:17.638056 1577 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 137 LEADER]: Stepping down as leader of term 137
I20260501 08:11:17.638095 1577 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 137 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:17.638290 1577 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 137.113, Last appended by leader: 113, Current term: 137, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.638432 1577 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 137 FOLLOWER]: Advancing to term 138
W20260501 08:11:17.639285 1575 consensus_queue.cc:1175] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2be24db2181749b9be4f7f93ba03e486" responder_term: 138 status { last_received { term: 137 index: 113 } last_committed_idx: 113 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 137. Current term is 138. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260501 08:11:17.702108 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 138 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 137 index: 113. Preceding OpId from leader: term: 138 index: 114. (index mismatch)
I20260501 08:11:17.702455 1568 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20260501 08:11:17.703734 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 138 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 137 index: 113. Preceding OpId from leader: term: 138 index: 114. (index mismatch)
I20260501 08:11:17.704180 1578 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20260501 08:11:17.805406 1574 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 138 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:17.805545 1574 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 138 FOLLOWER]: Advancing to term 139
I20260501 08:11:17.806381 1574 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.806564 1574 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 139 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:17.806756 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 139 candidate_status { last_received { term: 138 index: 114 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:17.806802 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 139 candidate_status { last_received { term: 138 index: 114 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:17.807060 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 139 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:17.807149 1574 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 139 FOLLOWER]: Leader election lost for term 139. Reason: could not achieve majority
I20260501 08:11:17.816704 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 139 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 138. Current term is 139. Ops: []
I20260501 08:11:17.816995 1569 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 138.114, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20260501 08:11:17.817134 1578 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 138 LEADER]: Stepping down as leader of term 138
I20260501 08:11:17.817194 1578 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 138 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:17.817281 1578 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 138.114, Last appended by leader: 114, Current term: 138, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.817471 1578 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 138 FOLLOWER]: Advancing to term 139
I20260501 08:11:17.906495 1575 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 138 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:17.906628 1575 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 138 FOLLOWER]: Advancing to term 139
I20260501 08:11:17.907485 1575 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.907720 1575 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 139 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:17.907965 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 139 candidate_status { last_received { term: 138 index: 114 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:17.908031 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 139 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 139: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:17.907999 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 139 candidate_status { last_received { term: 138 index: 114 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:17.908756 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 139 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 139.
I20260501 08:11:17.908988 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 139 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:17.909137 1575 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 139 FOLLOWER]: Leader election won for term 139
I20260501 08:11:17.909238 1575 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 139 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:17.909298 1575 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 138.114, Last appended by leader: 114, Current term: 139, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.909910 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 138 to 139, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 139 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:17.936635 1574 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 139 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:17.936741 1574 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 139 FOLLOWER]: Advancing to term 140
I20260501 08:11:17.937479 1574 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 140 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.937726 1574 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 140 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:17.937973 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 140 candidate_status { last_received { term: 138 index: 114 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:17.937973 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 140 candidate_status { last_received { term: 138 index: 114 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:17.938127 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 139 FOLLOWER]: Advancing to term 140
I20260501 08:11:17.938814 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 140 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 140.
I20260501 08:11:17.939050 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 140 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:17.939155 1574 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 140 FOLLOWER]: Leader election won for term 140
I20260501 08:11:17.939209 1574 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 140 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:17.939286 1574 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 138.114, Last appended by leader: 114, Current term: 140, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:17.939764 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 139 to 140, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 140 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:18.023381 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 140 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 139. Current term is 140. Ops: []
I20260501 08:11:18.023715 1575 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20260501 08:11:18.023887 1577 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 139 LEADER]: Stepping down as leader of term 139
I20260501 08:11:18.023927 1577 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 139 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:18.024035 1577 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 139.115, Last appended by leader: 115, Current term: 139, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.024117 1577 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 139 FOLLOWER]: Advancing to term 140
I20260501 08:11:18.025302 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 140 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 138 index: 114. Preceding OpId from leader: term: 140 index: 115. (index mismatch)
I20260501 08:11:18.025563 1574 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20260501 08:11:18.026563 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 140 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 139 index: 115. Preceding OpId from leader: term: 140 index: 115. (term mismatch)
I20260501 08:11:18.026614 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 114
I20260501 08:11:18.026638 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 139.115
I20260501 08:11:18.026664 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 140 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:18.026844 1574 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20260501 08:11:18.087212 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:18.127154 1578 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 140 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:18.127276 1578 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 140 FOLLOWER]: Advancing to term 141
I20260501 08:11:18.128063 1578 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.128317 1578 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 141 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:18.128559 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 141 candidate_status { last_received { term: 140 index: 115 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:18.128576 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 141 candidate_status { last_received { term: 140 index: 115 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:18.128810 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 141 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:18.128908 1578 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 141 FOLLOWER]: Leader election lost for term 141. Reason: could not achieve majority
I20260501 08:11:18.131541 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 141 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 140. Current term is 141. Ops: []
I20260501 08:11:18.131816 1574 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 140.115, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20260501 08:11:18.131923 1574 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 140 LEADER]: Stepping down as leader of term 140
I20260501 08:11:18.131942 1574 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 140 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:18.132007 1574 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 140.115, Last appended by leader: 115, Current term: 140, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.132086 1574 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 140 FOLLOWER]: Advancing to term 141
I20260501 08:11:18.231395 1578 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 141 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:18.231498 1578 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 141 FOLLOWER]: Advancing to term 142
I20260501 08:11:18.232333 1578 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 142 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.232518 1578 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 142 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:18.232749 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 142 candidate_status { last_received { term: 140 index: 115 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:18.232750 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 142 candidate_status { last_received { term: 140 index: 115 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:18.232928 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 140 FOLLOWER]: Advancing to term 142
I20260501 08:11:18.232867 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 141 FOLLOWER]: Advancing to term 142
I20260501 08:11:18.233527 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 142 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 142.
I20260501 08:11:18.233527 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 142 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 142.
I20260501 08:11:18.233831 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 142 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters:
I20260501 08:11:18.233981 1578 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 142 FOLLOWER]: Leader election won for term 142
I20260501 08:11:18.234040 1578 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 142 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:18.234083 1578 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 140.115, Last appended by leader: 115, Current term: 142, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.234618 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 140 to 142, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 142 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:18.314148 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 142 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 140 index: 115. Preceding OpId from leader: term: 142 index: 116. (index mismatch)
I20260501 08:11:18.314467 1578 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20260501 08:11:18.315698 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 142 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 140 index: 115. Preceding OpId from leader: term: 142 index: 116. (index mismatch)
I20260501 08:11:18.316011 1578 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20260501 08:11:18.514070 1574 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 142 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:18.514168 1574 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 142 FOLLOWER]: Advancing to term 143
I20260501 08:11:18.514868 1574 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.515057 1574 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 143 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:18.515285 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 143 candidate_status { last_received { term: 142 index: 116 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:18.515305 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 143 candidate_status { last_received { term: 142 index: 116 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:18.515558 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 143 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:18.515668 1574 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 143 FOLLOWER]: Leader election lost for term 143. Reason: could not achieve majority
I20260501 08:11:18.519176 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 143 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 142. Current term is 143. Ops: []
I20260501 08:11:18.519544 1569 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 142.116, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20260501 08:11:18.519698 1578 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 142 LEADER]: Stepping down as leader of term 142
I20260501 08:11:18.519784 1578 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 142 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:18.519869 1578 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 142.116, Last appended by leader: 116, Current term: 142, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.519975 1578 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 142 FOLLOWER]: Advancing to term 143
I20260501 08:11:18.607939 1575 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 142 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:18.608095 1575 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 142 FOLLOWER]: Advancing to term 143
I20260501 08:11:18.608814 1575 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.609051 1575 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 143 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:18.609282 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 143 candidate_status { last_received { term: 142 index: 116 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:18.609290 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 143 candidate_status { last_received { term: 142 index: 116 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:18.609484 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 143 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 143: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:18.609972 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 143 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 143.
I20260501 08:11:18.610165 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 143 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:18.610289 1575 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 143 FOLLOWER]: Leader election won for term 143
I20260501 08:11:18.610409 1575 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 143 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:18.610468 1575 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 142.116, Last appended by leader: 116, Current term: 143, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.610955 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 142 to 143, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 143 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:18.640157 1574 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 143 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:18.640247 1574 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 143 FOLLOWER]: Advancing to term 144
I20260501 08:11:18.640872 1574 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.641112 1574 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 144 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:18.641340 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 144 candidate_status { last_received { term: 142 index: 116 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:18.641346 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 144 candidate_status { last_received { term: 142 index: 116 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:18.641444 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 143 FOLLOWER]: Advancing to term 144
I20260501 08:11:18.641989 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 144 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 144.
I20260501 08:11:18.642203 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 144 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:18.642306 1574 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 144 FOLLOWER]: Leader election won for term 144
I20260501 08:11:18.642412 1574 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 144 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:18.642479 1574 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 142.116, Last appended by leader: 116, Current term: 144, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.642990 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 143 to 144, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 144 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:18.691645 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 144 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 143. Current term is 144. Ops: []
I20260501 08:11:18.692108 1575 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20260501 08:11:18.692281 1575 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 144.117 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 116
I20260501 08:11:18.692337 1580 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 143 LEADER]: Stepping down as leader of term 143
I20260501 08:11:18.692399 1580 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 143 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:18.692494 1580 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 143.117, Last appended by leader: 117, Current term: 143, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.692592 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 143 FOLLOWER]: Advancing to term 144
I20260501 08:11:18.730664 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 144 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 142 index: 116. Preceding OpId from leader: term: 144 index: 117. (index mismatch)
I20260501 08:11:18.731182 1574 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20260501 08:11:18.732478 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 144 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 143 index: 117. Preceding OpId from leader: term: 144 index: 117. (term mismatch)
I20260501 08:11:18.732532 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 116
I20260501 08:11:18.732550 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 143.117
I20260501 08:11:18.732568 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 144 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:18.732745 1582 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20260501 08:11:18.833215 1578 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 144 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:18.833348 1578 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 144 FOLLOWER]: Advancing to term 145
I20260501 08:11:18.834084 1578 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.834280 1578 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 145 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:18.834509 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 145 candidate_status { last_received { term: 144 index: 117 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:18.834509 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 145 candidate_status { last_received { term: 144 index: 117 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:18.834759 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 145 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:18.834868 1578 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 145 FOLLOWER]: Leader election lost for term 145. Reason: could not achieve majority
I20260501 08:11:18.844358 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 145 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 144. Current term is 145. Ops: []
I20260501 08:11:18.844638 1574 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 144.117, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20260501 08:11:18.844780 1582 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 144 LEADER]: Stepping down as leader of term 144
I20260501 08:11:18.844827 1582 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 144 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:18.844913 1582 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 144.117, Last appended by leader: 117, Current term: 144, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.845021 1582 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 144 FOLLOWER]: Advancing to term 145
I20260501 08:11:18.923180 1580 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 144 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:18.923296 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 144 FOLLOWER]: Advancing to term 145
I20260501 08:11:18.924085 1580 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.924347 1580 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 145 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:18.924573 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 145 candidate_status { last_received { term: 144 index: 117 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:18.924589 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 145 candidate_status { last_received { term: 144 index: 117 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:18.924763 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 145 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 145: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:18.925264 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 145 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 145.
I20260501 08:11:18.925478 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 145 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:18.925609 1580 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 145 FOLLOWER]: Leader election won for term 145
I20260501 08:11:18.925700 1580 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 145 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:18.925801 1580 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 144.117, Last appended by leader: 117, Current term: 145, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.926373 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 144 to 145, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 145 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:18.943924 1578 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 145 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:18.944051 1578 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 145 FOLLOWER]: Advancing to term 146
I20260501 08:11:18.944787 1578 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.944983 1578 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 146 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:18.945214 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 146 candidate_status { last_received { term: 144 index: 117 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:18.945232 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 146 candidate_status { last_received { term: 144 index: 117 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:18.945300 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 145 FOLLOWER]: Advancing to term 146
I20260501 08:11:18.945906 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 146 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 146.
I20260501 08:11:18.946121 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 146 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:18.946259 1578 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 146 FOLLOWER]: Leader election won for term 146
I20260501 08:11:18.946329 1578 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 146 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:18.946383 1578 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 144.117, Last appended by leader: 117, Current term: 146, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:18.947086 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 145 to 146, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 146 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:19.025588 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 146 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 144 index: 117. Preceding OpId from leader: term: 146 index: 118. (index mismatch)
I20260501 08:11:19.026016 1578 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20260501 08:11:19.027202 1233 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 145 LEADER]: Stepping down as leader of term 145
I20260501 08:11:19.027259 1233 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 145 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:19.027316 1233 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 145.118, Last appended by leader: 118, Current term: 145, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.027417 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 145 FOLLOWER]: Advancing to term 146
I20260501 08:11:19.028100 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 146 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 145 index: 118. Preceding OpId from leader: term: 146 index: 118. (term mismatch)
I20260501 08:11:19.028281 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 117
I20260501 08:11:19.028297 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 145.118
I20260501 08:11:19.028311 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 146 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:19.028553 1578 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20260501 08:11:19.087494 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:19.294353 1582 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 146 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:19.294480 1582 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 146 FOLLOWER]: Advancing to term 147
I20260501 08:11:19.295332 1582 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.295547 1582 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 147 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:19.295786 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 147 candidate_status { last_received { term: 146 index: 118 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:19.295785 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 147 candidate_status { last_received { term: 146 index: 118 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:19.296242 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 147 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:19.296348 1582 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 147 FOLLOWER]: Leader election lost for term 147. Reason: could not achieve majority
I20260501 08:11:19.297703 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 147 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 146. Current term is 147. Ops: []
I20260501 08:11:19.298009 1569 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 146.118, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20260501 08:11:19.298161 1569 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 146 LEADER]: Stepping down as leader of term 146
I20260501 08:11:19.298218 1569 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 146 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:19.298300 1569 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 146.118, Last appended by leader: 118, Current term: 146, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.298393 1569 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 146 FOLLOWER]: Advancing to term 147
I20260501 08:11:19.316479 1580 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 146 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:19.316596 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 146 FOLLOWER]: Advancing to term 147
I20260501 08:11:19.317356 1580 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.317554 1580 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 147 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:19.317780 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 147 candidate_status { last_received { term: 146 index: 118 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:19.317804 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 147 candidate_status { last_received { term: 146 index: 118 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:19.317915 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 147 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 147: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:19.318548 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 147 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 147.
I20260501 08:11:19.318745 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 147 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:19.318850 1580 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 147 FOLLOWER]: Leader election won for term 147
I20260501 08:11:19.318913 1580 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 147 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:19.318948 1580 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 146.118, Last appended by leader: 118, Current term: 147, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.319707 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 146 to 147, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 147 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:19.415233 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 147 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 146 index: 118. Preceding OpId from leader: term: 147 index: 119. (index mismatch)
I20260501 08:11:19.415594 1580 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20260501 08:11:19.416720 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 147 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 146 index: 118. Preceding OpId from leader: term: 147 index: 119. (index mismatch)
I20260501 08:11:19.416910 1580 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20260501 08:11:19.681596 1578 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 147 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:19.681715 1578 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 147 FOLLOWER]: Advancing to term 148
I20260501 08:11:19.682502 1578 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.682687 1578 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 148 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:19.682981 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 148 candidate_status { last_received { term: 147 index: 119 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:19.683007 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 148 candidate_status { last_received { term: 147 index: 119 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:19.683271 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 148 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:19.683413 1578 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 148 FOLLOWER]: Leader election lost for term 148. Reason: could not achieve majority
I20260501 08:11:19.685855 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 148 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 147. Current term is 148. Ops: []
I20260501 08:11:19.686255 1575 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 147.119, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20260501 08:11:19.686406 1580 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 147 LEADER]: Stepping down as leader of term 147
I20260501 08:11:19.686439 1580 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 147 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:19.686519 1580 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 147.119, Last appended by leader: 119, Current term: 147, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.686609 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 147 FOLLOWER]: Advancing to term 148
I20260501 08:11:19.701337 1582 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 147 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:19.701449 1582 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 147 FOLLOWER]: Advancing to term 148
I20260501 08:11:19.702164 1582 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.702356 1582 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 148 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:19.702605 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 148 candidate_status { last_received { term: 147 index: 119 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:19.702733 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 148 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 148: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:19.702592 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 148 candidate_status { last_received { term: 147 index: 119 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:19.703517 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 148 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 148.
I20260501 08:11:19.703763 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 148 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:19.703923 1582 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 148 FOLLOWER]: Leader election won for term 148
I20260501 08:11:19.704094 1582 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 148 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:19.704191 1582 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 147.119, Last appended by leader: 119, Current term: 148, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.704872 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 147 to 148, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 148 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:19.786543 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 148 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 147 index: 119. Preceding OpId from leader: term: 148 index: 120. (index mismatch)
I20260501 08:11:19.786872 1582 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20260501 08:11:19.788203 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 148 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 147 index: 119. Preceding OpId from leader: term: 148 index: 120. (index mismatch)
I20260501 08:11:19.788432 1584 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20260501 08:11:19.888688 1578 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 148 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:19.888777 1578 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 148 FOLLOWER]: Advancing to term 149
I20260501 08:11:19.889456 1578 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.889631 1578 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 149 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:19.889827 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 149 candidate_status { last_received { term: 148 index: 120 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:19.889827 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 149 candidate_status { last_received { term: 148 index: 120 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:19.890080 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 149 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:19.890184 1578 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 149 FOLLOWER]: Leader election lost for term 149. Reason: could not achieve majority
I20260501 08:11:19.910337 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 149 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 148. Current term is 149. Ops: []
I20260501 08:11:19.910670 1582 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 148.120, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20260501 08:11:19.910835 1582 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 148 LEADER]: Stepping down as leader of term 148
I20260501 08:11:19.910871 1582 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 148 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:19.910941 1582 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 148.120, Last appended by leader: 120, Current term: 148, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.911051 1582 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 148 FOLLOWER]: Advancing to term 149
I20260501 08:11:19.983582 1580 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 148 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:19.983691 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 148 FOLLOWER]: Advancing to term 149
I20260501 08:11:19.984635 1580 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.984818 1580 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 149 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:19.985061 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 149 candidate_status { last_received { term: 148 index: 120 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:19.985147 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 149 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 149: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:19.985075 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 149 candidate_status { last_received { term: 148 index: 120 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:19.985989 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 149 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 149.
I20260501 08:11:19.986251 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 149 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:19.986357 1580 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 149 FOLLOWER]: Leader election won for term 149
I20260501 08:11:19.986409 1580 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 149 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:19.986464 1580 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 148.120, Last appended by leader: 120, Current term: 149, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.987112 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 148 to 149, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 149 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:19.996574 1578 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 149 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:19.996683 1578 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 149 FOLLOWER]: Advancing to term 150
I20260501 08:11:19.997565 1578 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.997782 1578 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 150 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:19.998003 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 150 candidate_status { last_received { term: 148 index: 120 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:19.998045 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 150 candidate_status { last_received { term: 148 index: 120 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:19.998148 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 149 FOLLOWER]: Advancing to term 150
I20260501 08:11:19.998811 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 150 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 150.
I20260501 08:11:19.999032 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 150 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:19.999128 1578 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 150 FOLLOWER]: Leader election won for term 150
I20260501 08:11:19.999246 1578 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 150 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:19.999296 1578 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 148.120, Last appended by leader: 120, Current term: 150, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:19.999897 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 149 to 150, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 150 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:20.070545 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 150 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 149. Current term is 150. Ops: []
I20260501 08:11:20.070906 1580 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20260501 08:11:20.071064 1580 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 149 LEADER]: Stepping down as leader of term 149
I20260501 08:11:20.071105 1580 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 149 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:20.071197 1580 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 149.121, Last appended by leader: 121, Current term: 149, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.071273 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 149 FOLLOWER]: Advancing to term 150
I20260501 08:11:20.087823 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:20.093433 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 150 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 149 index: 121. Preceding OpId from leader: term: 150 index: 121. (term mismatch)
I20260501 08:11:20.093531 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 120
I20260501 08:11:20.093549 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 149.121
I20260501 08:11:20.093567 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 150 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:20.094007 1578 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20260501 08:11:20.095103 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 150 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 148 index: 120. Preceding OpId from leader: term: 150 index: 121. (index mismatch)
I20260501 08:11:20.095316 1578 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20260501 08:11:20.396477 1580 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 150 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:20.396590 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 150 FOLLOWER]: Advancing to term 151
I20260501 08:11:20.397305 1580 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.397490 1580 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 151 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:20.397719 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 151 candidate_status { last_received { term: 150 index: 121 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:20.397719 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 151 candidate_status { last_received { term: 150 index: 121 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:20.398077 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 151 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:20.398202 1580 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 151 FOLLOWER]: Leader election lost for term 151. Reason: could not achieve majority
I20260501 08:11:20.404505 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 151 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 150. Current term is 151. Ops: []
I20260501 08:11:20.404819 1585 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 150.121, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20260501 08:11:20.405023 1585 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 150 LEADER]: Stepping down as leader of term 150
I20260501 08:11:20.405082 1585 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 150 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:20.405160 1585 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 150.121, Last appended by leader: 121, Current term: 150, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.405272 1585 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 150 FOLLOWER]: Advancing to term 151
I20260501 08:11:20.452644 1582 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 150 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:20.452767 1582 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 150 FOLLOWER]: Advancing to term 151
I20260501 08:11:20.453436 1582 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.453661 1582 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 151 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:20.453934 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 151 candidate_status { last_received { term: 150 index: 121 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:20.454057 1233 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 151 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 151: Already voted for candidate da6eb0fe0b9f41dc9adc9114840069da in this term.
I20260501 08:11:20.453958 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 151 candidate_status { last_received { term: 150 index: 121 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:20.454716 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 151 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 151.
I20260501 08:11:20.454967 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 151 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:20.455073 1582 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 151 FOLLOWER]: Leader election won for term 151
I20260501 08:11:20.455128 1582 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 151 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:20.455204 1582 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 150.121, Last appended by leader: 121, Current term: 151, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.455729 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 150 to 151, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 151 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:20.536671 1580 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 151 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:20.536895 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 151 FOLLOWER]: Advancing to term 152
I20260501 08:11:20.537670 1580 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.537843 1580 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 152 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:20.538051 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 152 candidate_status { last_received { term: 150 index: 121 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:20.538116 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 152 candidate_status { last_received { term: 150 index: 121 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:20.538141 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 151 FOLLOWER]: Advancing to term 152
I20260501 08:11:20.538738 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 152 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 152.
I20260501 08:11:20.538978 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 152 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:20.539131 1580 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 152 FOLLOWER]: Leader election won for term 152
I20260501 08:11:20.539217 1580 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 152 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:20.539268 1580 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 150.121, Last appended by leader: 121, Current term: 152, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.539980 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 151 to 152, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 152 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:20.573557 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 152 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 151. Current term is 152. Ops: []
I20260501 08:11:20.573789 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 152 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 151. Current term is 152. Ops: []
I20260501 08:11:20.573884 1582 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20260501 08:11:20.574008 1582 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 151 LEADER]: Stepping down as leader of term 151
I20260501 08:11:20.574038 1582 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 151 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:20.574100 1582 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 151.122, Last appended by leader: 122, Current term: 151, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.574191 1584 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20260501 08:11:20.574236 1584 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Peer da6eb0fe0b9f41dc9adc9114840069da log is divergent from this leader: its last log entry 152.122 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 121
I20260501 08:11:20.574345 1582 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 151 FOLLOWER]: Advancing to term 152
I20260501 08:11:20.615377 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 152 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 151 index: 122. Preceding OpId from leader: term: 152 index: 122. (term mismatch)
I20260501 08:11:20.615459 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 121
I20260501 08:11:20.615473 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 151.122
I20260501 08:11:20.615489 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 152 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:20.615762 1580 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20260501 08:11:20.617144 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 152 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 150 index: 121. Preceding OpId from leader: term: 152 index: 122. (index mismatch)
I20260501 08:11:20.617419 1587 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20260501 08:11:20.718022 1585 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 152 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:20.718152 1585 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 152 FOLLOWER]: Advancing to term 153
I20260501 08:11:20.718997 1585 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.719192 1585 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 153 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:20.719442 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 153 candidate_status { last_received { term: 152 index: 122 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:20.719452 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 153 candidate_status { last_received { term: 152 index: 122 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:20.719702 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 153 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:20.719820 1585 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 153 FOLLOWER]: Leader election lost for term 153. Reason: could not achieve majority
I20260501 08:11:20.720561 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 153 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 152. Current term is 153. Ops: []
I20260501 08:11:20.720790 1575 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 152.122, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20260501 08:11:20.720907 1575 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 152 LEADER]: Stepping down as leader of term 152
I20260501 08:11:20.720942 1575 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 152 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:20.721010 1575 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 152.122, Last appended by leader: 122, Current term: 152, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.721093 1575 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 152 FOLLOWER]: Advancing to term 153
I20260501 08:11:20.816251 1582 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 152 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:20.816354 1582 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 152 FOLLOWER]: Advancing to term 153
I20260501 08:11:20.817063 1582 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.817267 1582 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 153 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:20.817462 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 153 candidate_status { last_received { term: 152 index: 122 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:20.817505 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 153 candidate_status { last_received { term: 152 index: 122 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:20.817580 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 153 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be24db2181749b9be4f7f93ba03e486 in current term 153: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:20.818210 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 153 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 153.
I20260501 08:11:20.818414 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 153 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:20.818542 1582 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 153 FOLLOWER]: Leader election won for term 153
I20260501 08:11:20.818595 1582 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 153 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:20.818655 1582 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 152.122, Last appended by leader: 122, Current term: 153, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.819159 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 152 to 153, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 153 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:20.839294 1585 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 153 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:20.839401 1585 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 153 FOLLOWER]: Advancing to term 154
I20260501 08:11:20.840111 1585 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.840332 1585 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 154 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:20.840575 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 154 candidate_status { last_received { term: 152 index: 122 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:20.840600 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 154 candidate_status { last_received { term: 152 index: 122 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:20.840675 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 153 FOLLOWER]: Advancing to term 154
I20260501 08:11:20.841312 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 154 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 154.
I20260501 08:11:20.841516 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 154 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:20.841625 1585 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 154 FOLLOWER]: Leader election won for term 154
I20260501 08:11:20.841709 1585 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 154 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:20.841761 1585 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 152.122, Last appended by leader: 122, Current term: 154, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.842263 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 153 to 154, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 154 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:20.918004 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 154 LEADER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 153. Current term is 154. Ops: []
I20260501 08:11:20.918340 1582 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20260501 08:11:20.918466 1582 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 154.123 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 122
I20260501 08:11:20.918509 1584 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 153 LEADER]: Stepping down as leader of term 153
I20260501 08:11:20.918557 1584 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 153 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:20.918639 1584 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 153.123, Last appended by leader: 123, Current term: 153, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:20.918720 1584 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 153 FOLLOWER]: Advancing to term 154
I20260501 08:11:20.932912 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 154 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 152 index: 122. Preceding OpId from leader: term: 154 index: 123. (index mismatch)
I20260501 08:11:20.933280 1585 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20260501 08:11:20.934321 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 154 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 153 index: 123. Preceding OpId from leader: term: 154 index: 123. (term mismatch)
I20260501 08:11:20.934374 1384 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting all ops after (but not including) 122
I20260501 08:11:20.934394 1384 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: Aborting uncommitted NO_OP operation due to leader change: 153.123
I20260501 08:11:20.934412 1384 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 154 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:20.934576 1585 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20260501 08:11:21.035151 1575 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 154 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:21.035260 1575 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 154 FOLLOWER]: Advancing to term 155
I20260501 08:11:21.035956 1575 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.036187 1575 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 155 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:21.036409 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 155 candidate_status { last_received { term: 154 index: 123 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:21.036417 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 155 candidate_status { last_received { term: 154 index: 123 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:21.036500 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 154 FOLLOWER]: Advancing to term 155
I20260501 08:11:21.036584 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 155 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 154. Current term is 155. Ops: []
I20260501 08:11:21.036813 1585 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 154.123, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20260501 08:11:21.037006 1585 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 154 LEADER]: Stepping down as leader of term 154
I20260501 08:11:21.037047 1585 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 154 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:21.037101 1585 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 154.123, Last appended by leader: 123, Current term: 154, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.037182 1585 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 154 FOLLOWER]: Advancing to term 155
I20260501 08:11:21.037325 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 155 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 155.
I20260501 08:11:21.037528 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 155 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:21.037637 1575 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 155 FOLLOWER]: Leader election won for term 155
I20260501 08:11:21.037707 1575 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 155 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:21.037760 1575 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 154.123, Last appended by leader: 123, Current term: 155, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.038362 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 154 to 155, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 155 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:21.088109 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:21.131942 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 155 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 154 index: 123. Preceding OpId from leader: term: 155 index: 124. (index mismatch)
I20260501 08:11:21.132289 1575 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20260501 08:11:21.133533 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 155 FOLLOWER]: Refusing update from remote peer da6eb0fe0b9f41dc9adc9114840069da: Log matching property violated. Preceding OpId in replica: term: 154 index: 123. Preceding OpId from leader: term: 155 index: 124. (index mismatch)
I20260501 08:11:21.133726 1580 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20260501 08:11:21.235174 1582 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 155 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:21.235316 1582 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 155 FOLLOWER]: Advancing to term 156
I20260501 08:11:21.235986 1582 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.236284 1582 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 156 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:21.236492 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 156 candidate_status { last_received { term: 155 index: 124 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:21.236583 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 156 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 155. Current term is 156. Ops: []
I20260501 08:11:21.236584 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 156 candidate_status { last_received { term: 155 index: 124 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:21.236876 1580 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 155.124, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20260501 08:11:21.236886 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 156 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:21.237026 1580 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 155 LEADER]: Stepping down as leader of term 155
I20260501 08:11:21.237059 1580 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 155 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:21.237134 1582 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 156 FOLLOWER]: Leader election lost for term 156. Reason: could not achieve majority
I20260501 08:11:21.237169 1580 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 155.124, Last appended by leader: 124, Current term: 155, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.237267 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 155 FOLLOWER]: Advancing to term 156
I20260501 08:11:21.318300 1585 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 155 FOLLOWER]: Starting leader election (detected failure of leader da6eb0fe0b9f41dc9adc9114840069da)
I20260501 08:11:21.318411 1585 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 155 FOLLOWER]: Advancing to term 156
I20260501 08:11:21.319236 1585 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.319451 1585 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 156 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:21.319694 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 156 candidate_status { last_received { term: 155 index: 124 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:21.319796 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 156 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b9150123ff048d08f1d65634bf2425a in current term 156: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:21.319744 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 156 candidate_status { last_received { term: 155 index: 124 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:21.320549 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 156 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 156.
I20260501 08:11:21.320785 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 156 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:21.320904 1585 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 156 FOLLOWER]: Leader election won for term 156
I20260501 08:11:21.321013 1585 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 156 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:21.321074 1585 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 155.124, Last appended by leader: 124, Current term: 156, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.321585 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 155 to 156, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 156 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:21.374428 1582 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 156 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:21.374545 1582 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 156 FOLLOWER]: Advancing to term 157
I20260501 08:11:21.375198 1582 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.375507 1582 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 157 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:21.375626 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 157 candidate_status { last_received { term: 155 index: 124 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:21.375665 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 157 candidate_status { last_received { term: 155 index: 124 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:21.375718 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 156 FOLLOWER]: Advancing to term 157
I20260501 08:11:21.376507 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 157 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 157.
I20260501 08:11:21.376725 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 157 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:21.376828 1582 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 157 FOLLOWER]: Leader election won for term 157
I20260501 08:11:21.376922 1582 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 157 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:21.376983 1582 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 155.124, Last appended by leader: 124, Current term: 157, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.377555 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 156 to 157, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 157 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:21.397437 1233 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 157 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 156. Current term is 157. Ops: []
I20260501 08:11:21.397915 1585 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: INVALID_TERM, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20260501 08:11:21.398065 1585 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 156 LEADER]: Stepping down as leader of term 156
I20260501 08:11:21.398126 1585 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 156 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:21.398205 1585 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 156.125, Last appended by leader: 125, Current term: 156, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.398293 1585 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 156 FOLLOWER]: Advancing to term 157
I20260501 08:11:21.467566 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 157 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 155 index: 124. Preceding OpId from leader: term: 157 index: 125. (index mismatch)
I20260501 08:11:21.467849 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 157 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 156 index: 125. Preceding OpId from leader: term: 157 index: 125. (term mismatch)
I20260501 08:11:21.467907 1310 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting all ops after (but not including) 124
I20260501 08:11:21.467875 1582 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20260501 08:11:21.467941 1310 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: Aborting uncommitted NO_OP operation due to leader change: 156.125
I20260501 08:11:21.468048 1310 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 157 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:21.468273 1582 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20260501 08:11:21.671238 1585 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 157 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:21.671350 1585 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 157 FOLLOWER]: Advancing to term 158
I20260501 08:11:21.673221 1585 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.673422 1585 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 158 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:21.673708 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 158 candidate_status { last_received { term: 157 index: 125 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:21.673713 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 158 candidate_status { last_received { term: 157 index: 125 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:21.673955 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 158 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:21.674073 1585 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 158 FOLLOWER]: Leader election lost for term 158. Reason: could not achieve majority
I20260501 08:11:21.686563 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 158 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 157. Current term is 158. Ops: []
I20260501 08:11:21.686836 1589 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 157.125, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20260501 08:11:21.686985 1589 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 157 LEADER]: Stepping down as leader of term 157
I20260501 08:11:21.687044 1589 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 157 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:21.687175 1589 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 157.125, Last appended by leader: 125, Current term: 157, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.687327 1589 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 157 FOLLOWER]: Advancing to term 158
I20260501 08:11:21.762655 1580 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 157 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:21.762907 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 157 FOLLOWER]: Advancing to term 158
I20260501 08:11:21.763713 1580 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.763962 1580 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 158 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:21.764235 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 158 candidate_status { last_received { term: 157 index: 125 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:21.764321 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 158 candidate_status { last_received { term: 157 index: 125 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:21.764467 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 158 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 158: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:21.765002 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 158 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 158.
I20260501 08:11:21.765210 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 158 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:21.765311 1580 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 158 FOLLOWER]: Leader election won for term 158
I20260501 08:11:21.765424 1580 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 158 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:21.765496 1580 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 157.125, Last appended by leader: 125, Current term: 158, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.766172 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 157 to 158, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 158 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:21.823000 1585 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 158 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:21.823105 1585 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 158 FOLLOWER]: Advancing to term 159
I20260501 08:11:21.823776 1585 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.824052 1585 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 159 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:21.824281 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 159 candidate_status { last_received { term: 157 index: 125 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:21.824290 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 159 candidate_status { last_received { term: 157 index: 125 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:21.824514 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 158 FOLLOWER]: Advancing to term 159
I20260501 08:11:21.825098 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 159 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 159.
I20260501 08:11:21.825300 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 159 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:21.825394 1585 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 159 FOLLOWER]: Leader election won for term 159
I20260501 08:11:21.825449 1585 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 159 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:21.825491 1585 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 157.125, Last appended by leader: 125, Current term: 159, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.825950 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 158 to 159, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 159 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:21.846539 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 159 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 158. Current term is 159. Ops: []
I20260501 08:11:21.846974 1580 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20260501 08:11:21.847112 1580 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 4b9150123ff048d08f1d65634bf2425a log is divergent from this leader: its last log entry 159.126 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 125
I20260501 08:11:21.847154 1590 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 158 LEADER]: Stepping down as leader of term 158
I20260501 08:11:21.847201 1590 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 158 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:21.847313 1590 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 158.126, Last appended by leader: 126, Current term: 158, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:21.847407 1590 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 158 FOLLOWER]: Advancing to term 159
I20260501 08:11:21.926512 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 159 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 158 index: 126. Preceding OpId from leader: term: 159 index: 126. (term mismatch)
I20260501 08:11:21.926641 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 125
I20260501 08:11:21.926664 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 158.126
I20260501 08:11:21.926687 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 159 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:21.926957 1585 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20260501 08:11:21.927973 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 159 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 157 index: 125. Preceding OpId from leader: term: 159 index: 126. (index mismatch)
I20260501 08:11:21.928259 1586 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20260501 08:11:22.029654 1589 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 159 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:22.029762 1589 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 159 FOLLOWER]: Advancing to term 160
I20260501 08:11:22.030447 1589 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.030637 1589 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 160 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:22.030848 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 160 candidate_status { last_received { term: 159 index: 126 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:22.030879 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 160 candidate_status { last_received { term: 159 index: 126 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:22.031104 1349 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 160 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:22.031205 1589 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 160 FOLLOWER]: Leader election lost for term 160. Reason: could not achieve majority
I20260501 08:11:22.048749 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 160 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 159. Current term is 160. Ops: []
I20260501 08:11:22.049077 1586 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 159.126, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20260501 08:11:22.049206 1586 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 159 LEADER]: Stepping down as leader of term 159
I20260501 08:11:22.049247 1586 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 159 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:22.049322 1586 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 159.126, Last appended by leader: 126, Current term: 159, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.049404 1586 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 159 FOLLOWER]: Advancing to term 160
I20260501 08:11:22.088413 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:22.104825 1590 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 159 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:22.104930 1590 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 159 FOLLOWER]: Advancing to term 160
I20260501 08:11:22.105703 1590 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.105897 1590 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 160 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:22.106146 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 160 candidate_status { last_received { term: 159 index: 126 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:22.106163 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 160 candidate_status { last_received { term: 159 index: 126 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:22.106235 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 160 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 160: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:22.106797 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 160 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 160.
I20260501 08:11:22.107056 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 160 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:22.107152 1590 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 160 FOLLOWER]: Leader election won for term 160
I20260501 08:11:22.107250 1590 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 160 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:22.107316 1590 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 159.126, Last appended by leader: 126, Current term: 160, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.107774 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 159 to 160, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 160 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:22.170822 1589 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 160 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:22.170940 1589 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 160 FOLLOWER]: Advancing to term 161
I20260501 08:11:22.171674 1589 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.171870 1589 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 161 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:22.172164 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 161 candidate_status { last_received { term: 159 index: 126 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:22.172185 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 161 candidate_status { last_received { term: 159 index: 126 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:22.172263 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 160 FOLLOWER]: Advancing to term 161
I20260501 08:11:22.172935 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 161 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 161.
I20260501 08:11:22.173127 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 161 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:22.173223 1589 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 161 FOLLOWER]: Leader election won for term 161
I20260501 08:11:22.173278 1589 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 161 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:22.173336 1589 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 159.126, Last appended by leader: 126, Current term: 161, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.173983 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 160 to 161, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 161 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:22.219509 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 161 LEADER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 160. Current term is 161. Ops: []
I20260501 08:11:22.219925 1590 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20260501 08:11:22.220084 1590 consensus_queue.cc:1243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer 2be24db2181749b9be4f7f93ba03e486 log is divergent from this leader: its last log entry 161.127 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 126
I20260501 08:11:22.220181 1580 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 160 LEADER]: Stepping down as leader of term 160
I20260501 08:11:22.220247 1580 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 160 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:22.220321 1580 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 160.127, Last appended by leader: 127, Current term: 160, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.220402 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 160 FOLLOWER]: Advancing to term 161
I20260501 08:11:22.251456 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 161 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 159 index: 126. Preceding OpId from leader: term: 161 index: 127. (index mismatch)
I20260501 08:11:22.251732 1589 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20260501 08:11:22.252782 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 161 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 160 index: 127. Preceding OpId from leader: term: 161 index: 127. (term mismatch)
I20260501 08:11:22.252842 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 126
I20260501 08:11:22.252866 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 160.127
I20260501 08:11:22.252887 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 161 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:22.253194 1592 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20260501 08:11:22.451501 1586 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 161 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:22.451617 1586 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 161 FOLLOWER]: Advancing to term 162
I20260501 08:11:22.452648 1586 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 162 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.452953 1586 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 162 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:22.453083 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 162 candidate_status { last_received { term: 161 index: 127 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:22.453210 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 162 candidate_status { last_received { term: 161 index: 127 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:22.453469 1275 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 162 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:22.453584 1586 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 162 FOLLOWER]: Leader election lost for term 162. Reason: could not achieve majority
I20260501 08:11:22.460075 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 162 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 161. Current term is 162. Ops: []
I20260501 08:11:22.460433 1582 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 161.127, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20260501 08:11:22.460578 1589 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 161 LEADER]: Stepping down as leader of term 161
I20260501 08:11:22.460620 1589 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 161 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:22.460700 1589 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 161.127, Last appended by leader: 127, Current term: 161, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.460790 1589 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 161 FOLLOWER]: Advancing to term 162
I20260501 08:11:22.560556 1580 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 161 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:22.560669 1580 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 161 FOLLOWER]: Advancing to term 162
I20260501 08:11:22.561043 1586 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 162 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:22.561110 1586 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 162 FOLLOWER]: Advancing to term 163
I20260501 08:11:22.561482 1580 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 162 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.561690 1580 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 162 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:22.561770 1586 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 163 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.561990 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 162 candidate_status { last_received { term: 161 index: 127 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:22.562026 1586 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 163 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:22.561894 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 162 candidate_status { last_received { term: 161 index: 127 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:22.562069 1310 raft_consensus.cc:2368] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da for earlier term 162. Current term is 163.
I20260501 08:11:22.562265 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 163 candidate_status { last_received { term: 161 index: 127 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:22.562332 1233 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 162 FOLLOWER]: Advancing to term 163
I20260501 08:11:22.562252 1197 leader_election.cc:400] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 162 election: Vote denied by peer 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689) with higher term. Message: Invalid argument: T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da for earlier term 162. Current term is 163.
I20260501 08:11:22.562417 1197 leader_election.cc:403] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 162 election: Cancelling election due to peer responding with higher term
I20260501 08:11:22.562237 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 163 candidate_status { last_received { term: 161 index: 127 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:22.562825 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 162 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 162.
I20260501 08:11:22.563027 1233 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 163 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b9150123ff048d08f1d65634bf2425a in term 163.
I20260501 08:11:22.563117 1580 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 163 FOLLOWER]: Leader election lost for term 162. Reason: Vote denied by peer 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689) with higher term. Message: Invalid argument: T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da for earlier term 162. Current term is 163.
I20260501 08:11:22.563220 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 163 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:22.563330 1586 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 163 FOLLOWER]: Leader election won for term 163
I20260501 08:11:22.563469 1586 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 163 LEADER]: Becoming Leader. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:22.563541 1586 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 161.127, Last appended by leader: 127, Current term: 163, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.563992 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a reported cstate change: term changed from 161 to 163, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to 4b9150123ff048d08f1d65634bf2425a (127.1.19.2). New cstate: current_term: 163 leader_uuid: "4b9150123ff048d08f1d65634bf2425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:22.643380 1384 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 162 FOLLOWER]: Advancing to term 163
I20260501 08:11:22.644212 1384 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 163 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 161 index: 127. Preceding OpId from leader: term: 163 index: 128. (index mismatch)
I20260501 08:11:22.644512 1586 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20260501 08:11:22.645812 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 163 FOLLOWER]: Refusing update from remote peer 4b9150123ff048d08f1d65634bf2425a: Log matching property violated. Preceding OpId in replica: term: 161 index: 127. Preceding OpId from leader: term: 163 index: 128. (index mismatch)
I20260501 08:11:22.646006 1586 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20260501 08:11:22.746270 1589 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 163 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:22.746368 1589 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 163 FOLLOWER]: Advancing to term 164
I20260501 08:11:22.747078 1589 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 164 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.747262 1589 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 164 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:22.747476 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 164 candidate_status { last_received { term: 163 index: 128 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:22.747476 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 164 candidate_status { last_received { term: 163 index: 128 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:22.747784 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 164 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486; no voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:22.748021 1589 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 164 FOLLOWER]: Leader election lost for term 164. Reason: could not achieve majority
I20260501 08:11:22.751914 1384 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 164 FOLLOWER]: Rejecting Update request from peer 4b9150123ff048d08f1d65634bf2425a for earlier term 163. Current term is 164. Ops: []
I20260501 08:11:22.752456 1593 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 }, Status: INVALID_TERM, Last received: 163.128, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20260501 08:11:22.752581 1593 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 163 LEADER]: Stepping down as leader of term 163
I20260501 08:11:22.752600 1593 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 163 LEADER]: Becoming Follower/Learner. State: Replica: 4b9150123ff048d08f1d65634bf2425a, State: Running, Role: LEADER
I20260501 08:11:22.752677 1593 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 163.128, Last appended by leader: 128, Current term: 163, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.752758 1593 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 163 FOLLOWER]: Advancing to term 164
I20260501 08:11:22.846624 1590 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 163 FOLLOWER]: Starting leader election (detected failure of leader 4b9150123ff048d08f1d65634bf2425a)
I20260501 08:11:22.846719 1590 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 163 FOLLOWER]: Advancing to term 164
I20260501 08:11:22.847425 1590 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 164 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.847658 1590 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 164 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:22.847848 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 164 candidate_status { last_received { term: 163 index: 128 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:22.847841 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 164 candidate_status { last_received { term: 163 index: 128 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:22.848016 1384 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 164 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 164: Already voted for candidate 2be24db2181749b9be4f7f93ba03e486 in this term.
I20260501 08:11:22.848608 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 164 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 164.
I20260501 08:11:22.848819 1197 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 164 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a, da6eb0fe0b9f41dc9adc9114840069da; no voters: 2be24db2181749b9be4f7f93ba03e486
I20260501 08:11:22.848935 1590 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 164 FOLLOWER]: Leader election won for term 164
I20260501 08:11:22.849035 1590 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 164 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:22.849086 1590 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 163.128, Last appended by leader: 128, Current term: 164, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.849573 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 163 to 164, leader changed from 4b9150123ff048d08f1d65634bf2425a (127.1.19.2) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 164 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:22.858498 1589 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 164 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:22.858639 1589 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 164 FOLLOWER]: Advancing to term 165
I20260501 08:11:22.859328 1589 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 165 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.859545 1589 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 165 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:22.859776 1233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 165 candidate_status { last_received { term: 163 index: 128 } } ignore_live_leader: false dest_uuid: "da6eb0fe0b9f41dc9adc9114840069da"
I20260501 08:11:22.859835 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "2be24db2181749b9be4f7f93ba03e486" candidate_term: 165 candidate_status { last_received { term: 163 index: 128 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:22.859891 1310 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 164 FOLLOWER]: Advancing to term 165
I20260501 08:11:22.860569 1310 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 165 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be24db2181749b9be4f7f93ba03e486 in term 165.
I20260501 08:11:22.860778 1348 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [CANDIDATE]: Term 165 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, 4b9150123ff048d08f1d65634bf2425a; no voters: da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:22.860885 1589 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 165 FOLLOWER]: Leader election won for term 165
I20260501 08:11:22.861017 1589 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 165 LEADER]: Becoming Leader. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:22.861081 1589 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 163.128, Last appended by leader: 128, Current term: 165, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.861631 1130 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 reported cstate change: term changed from 164 to 165, leader changed from da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1) to 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3). New cstate: current_term: 165 leader_uuid: "2be24db2181749b9be4f7f93ba03e486" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:22.947974 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 165 FOLLOWER]: Rejecting Update request from peer da6eb0fe0b9f41dc9adc9114840069da for earlier term 164. Current term is 165. Ops: []
I20260501 08:11:22.948364 1590 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20260501 08:11:22.948503 1590 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 164 LEADER]: Stepping down as leader of term 164
I20260501 08:11:22.948540 1590 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 164 LEADER]: Becoming Follower/Learner. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:22.948608 1590 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 164.129, Last appended by leader: 129, Current term: 164, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:22.948681 1590 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 164 FOLLOWER]: Advancing to term 165
I20260501 08:11:22.950994 1310 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 165 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 163 index: 128. Preceding OpId from leader: term: 165 index: 129. (index mismatch)
I20260501 08:11:22.951323 1589 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20260501 08:11:22.952580 1233 raft_consensus.cc:1275] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 165 FOLLOWER]: Refusing update from remote peer 2be24db2181749b9be4f7f93ba03e486: Log matching property violated. Preceding OpId in replica: term: 164 index: 129. Preceding OpId from leader: term: 165 index: 129. (term mismatch)
I20260501 08:11:22.952641 1233 pending_rounds.cc:85] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting all ops after (but not including) 128
I20260501 08:11:22.952662 1233 pending_rounds.cc:107] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting uncommitted NO_OP operation due to leader change: 164.129
I20260501 08:11:22.952684 1233 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 165 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:22.952883 1589 consensus_queue.cc:1048] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20260501 08:11:23.088757 1100 timestamp_advancement-itest.cc:267] GCing logs...
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:267: Failure
Expected: (gcable_size) > (0), actual: 0 vs 0
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:332: Failure
Expected: GCUntilNoWritesInWAL(ts, replica) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260501 08:11:23.089201 1100 tablet_server.cc:179] TabletServer@127.1.19.1:0 shutting down...
I20260501 08:11:23.134650 1593 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 165 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:23.134797 1593 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 165 FOLLOWER]: Advancing to term 166
I20260501 08:11:23.135525 1593 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 166 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:23.135751 1593 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 166 election: Requested vote from peers 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049), da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841)
I20260501 08:11:23.136018 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "4b9150123ff048d08f1d65634bf2425a" candidate_term: 166 candidate_status { last_received { term: 165 index: 129 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
W20260501 08:11:23.136083 1274 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260501 08:11:23.136572 1274 leader_election.cc:336] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 166 election: RPC error from VoteRequest() call to peer da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1:36841): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260501 08:11:23.136662 1274 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [CANDIDATE]: Term 166 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b9150123ff048d08f1d65634bf2425a; no voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da
I20260501 08:11:23.136736 1593 raft_consensus.cc:2749] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 166 FOLLOWER]: Leader election lost for term 166. Reason: could not achieve majority
I20260501 08:11:23.140745 1310 raft_consensus.cc:1240] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 166 FOLLOWER]: Rejecting Update request from peer 2be24db2181749b9be4f7f93ba03e486 for earlier term 165. Current term is 166. Ops: []
I20260501 08:11:23.141063 1595 consensus_queue.cc:1059] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 }, Status: INVALID_TERM, Last received: 165.129, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20260501 08:11:23.141208 1582 raft_consensus.cc:3055] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 165 LEADER]: Stepping down as leader of term 165
I20260501 08:11:23.141249 1582 raft_consensus.cc:740] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 165 LEADER]: Becoming Follower/Learner. State: Replica: 2be24db2181749b9be4f7f93ba03e486, State: Running, Role: LEADER
I20260501 08:11:23.141315 1582 consensus_queue.cc:260] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 165.129, Last appended by leader: 129, Current term: 165, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:23.141395 1582 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 165 FOLLOWER]: Advancing to term 166
I20260501 08:11:23.174768 1590 raft_consensus.cc:493] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 165 FOLLOWER]: Starting leader election (detected failure of leader 2be24db2181749b9be4f7f93ba03e486)
I20260501 08:11:23.174885 1590 raft_consensus.cc:3060] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 165 FOLLOWER]: Advancing to term 166
I20260501 08:11:23.175565 1590 raft_consensus.cc:515] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 166 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:23.175765 1590 leader_election.cc:290] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 166 election: Requested vote from peers 4b9150123ff048d08f1d65634bf2425a (127.1.19.2:43689), 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3:46049)
I20260501 08:11:23.176000 1310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 166 candidate_status { last_received { term: 165 index: 129 } } ignore_live_leader: false dest_uuid: "4b9150123ff048d08f1d65634bf2425a"
I20260501 08:11:23.176070 1310 raft_consensus.cc:2393] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 166 FOLLOWER]: Leader election vote request: Denying vote to candidate da6eb0fe0b9f41dc9adc9114840069da in current term 166: Already voted for candidate 4b9150123ff048d08f1d65634bf2425a in this term.
I20260501 08:11:23.176013 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d0191c11e464652ac0b4901a8d0fd06" candidate_uuid: "da6eb0fe0b9f41dc9adc9114840069da" candidate_term: 166 candidate_status { last_received { term: 165 index: 129 } } ignore_live_leader: false dest_uuid: "2be24db2181749b9be4f7f93ba03e486"
I20260501 08:11:23.176784 1384 raft_consensus.cc:2468] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 166 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6eb0fe0b9f41dc9adc9114840069da in term 166.
I20260501 08:11:23.177064 1199 leader_election.cc:304] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [CANDIDATE]: Term 166 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be24db2181749b9be4f7f93ba03e486, da6eb0fe0b9f41dc9adc9114840069da; no voters: 4b9150123ff048d08f1d65634bf2425a
I20260501 08:11:23.177219 1590 raft_consensus.cc:2804] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 166 FOLLOWER]: Leader election won for term 166
I20260501 08:11:23.177287 1590 raft_consensus.cc:697] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 166 LEADER]: Becoming Leader. State: Replica: da6eb0fe0b9f41dc9adc9114840069da, State: Running, Role: LEADER
I20260501 08:11:23.177346 1590 consensus_queue.cc:237] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 165.129, Last appended by leader: 129, Current term: 166, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } }
I20260501 08:11:23.177843 1127 catalog_manager.cc:5671] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da reported cstate change: term changed from 165 to 166, leader changed from 2be24db2181749b9be4f7f93ba03e486 (127.1.19.3) to da6eb0fe0b9f41dc9adc9114840069da (127.1.19.1). New cstate: current_term: 166 leader_uuid: "da6eb0fe0b9f41dc9adc9114840069da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b9150123ff048d08f1d65634bf2425a" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be24db2181749b9be4f7f93ba03e486" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6eb0fe0b9f41dc9adc9114840069da" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 36841 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:23.229059 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:23.229259 1100 tablet_replica.cc:333] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: stopping tablet replica
I20260501 08:11:23.229352 1100 raft_consensus.cc:2243] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 166 LEADER]: Raft consensus shutting down.
I20260501 08:11:23.229441 1100 pending_rounds.cc:70] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Trying to abort 1 pending ops.
I20260501 08:11:23.229471 1100 pending_rounds.cc:77] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da: Aborting op as it isn't in flight: id { term: 166 index: 130 } timestamp: 7281144148695060480 op_type: NO_OP noop_request { }
I20260501 08:11:23.229542 1100 raft_consensus.cc:2889] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 166 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260501 08:11:23.229575 1100 raft_consensus.cc:2272] T 4d0191c11e464652ac0b4901a8d0fd06 P da6eb0fe0b9f41dc9adc9114840069da [term 166 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:23.242449 1100 tablet_server.cc:196] TabletServer@127.1.19.1:0 shutdown complete.
I20260501 08:11:23.243908 1100 tablet_server.cc:179] TabletServer@127.1.19.2:0 shutting down...
I20260501 08:11:23.272917 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:23.273082 1100 tablet_replica.cc:333] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a: stopping tablet replica
I20260501 08:11:23.273144 1100 raft_consensus.cc:2243] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 166 FOLLOWER]: Raft consensus shutting down.
I20260501 08:11:23.273233 1100 raft_consensus.cc:2272] T 4d0191c11e464652ac0b4901a8d0fd06 P 4b9150123ff048d08f1d65634bf2425a [term 166 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:23.285771 1100 tablet_server.cc:196] TabletServer@127.1.19.2:0 shutdown complete.
I20260501 08:11:23.286862 1100 tablet_server.cc:179] TabletServer@127.1.19.3:0 shutting down...
I20260501 08:11:23.295610 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:23.295742 1100 tablet_replica.cc:333] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486: stopping tablet replica
I20260501 08:11:23.295802 1100 raft_consensus.cc:2243] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 166 FOLLOWER]: Raft consensus shutting down.
I20260501 08:11:23.295861 1100 raft_consensus.cc:2272] T 4d0191c11e464652ac0b4901a8d0fd06 P 2be24db2181749b9be4f7f93ba03e486 [term 166 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:23.309211 1100 tablet_server.cc:196] TabletServer@127.1.19.3:0 shutdown complete.
I20260501 08:11:23.310319 1100 master.cc:562] Master@127.1.19.62:45747 shutting down...
I20260501 08:11:23.398191 1100 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 1 LEADER]: Raft consensus shutting down.
I20260501 08:11:23.398308 1100 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:23.398355 1100 tablet_replica.cc:333] T 00000000000000000000000000000000 P 42b281f3ef9c42f6a07d3ec8932cf8f8: stopping tablet replica
I20260501 08:11:23.410583 1100 master.cc:584] Master@127.1.19.62:45747 shutdown complete.
I20260501 08:11:23.412760 1100 test_util.cc:182] -----------------------------------------------
I20260501 08:11:23.412822 1100 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1777623050595939-1100-0
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData (32812 ms)
[ RUN ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap
I20260501 08:11:23.413370 1100 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.19.62:39201
I20260501 08:11:23.413539 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:23.415086 1598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:23.415096 1600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:23.415324 1100 server_base.cc:1061] running on GCE node
W20260501 08:11:23.415350 1597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:23.415462 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:23.415635 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:23.415772 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623083415772 us; error 0 us; skew 500 ppm
I20260501 08:11:23.416479 1100 webserver.cc:492] Webserver started at http://127.1.19.62:41623/ using document root <none> and password file <none>
I20260501 08:11:23.416553 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:23.416587 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:23.416623 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:23.416926 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/master-0-root/instance:
uuid: "f3ec594bf2fe433884fb71a197c136a1"
format_stamp: "Formatted at 2026-05-01 08:11:23 on dist-test-slave-pv7w"
I20260501 08:11:23.417918 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260501 08:11:23.418491 1605 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.418669 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.418733 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/master-0-root
uuid: "f3ec594bf2fe433884fb71a197c136a1"
format_stamp: "Formatted at 2026-05-01 08:11:23 on dist-test-slave-pv7w"
I20260501 08:11:23.418816 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:23.426630 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:23.426921 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:23.430334 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.62:39201
I20260501 08:11:23.432498 1657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.62:39201 every 8 connection(s)
I20260501 08:11:23.432650 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:11:23.433833 1658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1: Bootstrap starting.
I20260501 08:11:23.434182 1658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1: Neither blocks nor log segments found. Creating new log.
I20260501 08:11:23.434712 1658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1: No bootstrap required, opened a new log
I20260501 08:11:23.434867 1658 raft_consensus.cc:359] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ec594bf2fe433884fb71a197c136a1" member_type: VOTER }
I20260501 08:11:23.434921 1658 raft_consensus.cc:385] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:11:23.434937 1658 raft_consensus.cc:740] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ec594bf2fe433884fb71a197c136a1, State: Initialized, Role: FOLLOWER
I20260501 08:11:23.434973 1658 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ec594bf2fe433884fb71a197c136a1" member_type: VOTER }
I20260501 08:11:23.434998 1658 raft_consensus.cc:399] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 08:11:23.435011 1658 raft_consensus.cc:493] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 08:11:23.435025 1658 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:11:23.435614 1658 raft_consensus.cc:515] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ec594bf2fe433884fb71a197c136a1" member_type: VOTER }
I20260501 08:11:23.435670 1658 leader_election.cc:304] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3ec594bf2fe433884fb71a197c136a1; no voters:
I20260501 08:11:23.435801 1658 leader_election.cc:290] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 08:11:23.435899 1661 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 08:11:23.435994 1658 sys_catalog.cc:565] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 08:11:23.436058 1661 raft_consensus.cc:697] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 1 LEADER]: Becoming Leader. State: Replica: f3ec594bf2fe433884fb71a197c136a1, State: Running, Role: LEADER
I20260501 08:11:23.436156 1661 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ec594bf2fe433884fb71a197c136a1" member_type: VOTER }
I20260501 08:11:23.436452 1662 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3ec594bf2fe433884fb71a197c136a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ec594bf2fe433884fb71a197c136a1" member_type: VOTER } }
I20260501 08:11:23.436512 1663 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3ec594bf2fe433884fb71a197c136a1. Latest consensus state: current_term: 1 leader_uuid: "f3ec594bf2fe433884fb71a197c136a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ec594bf2fe433884fb71a197c136a1" member_type: VOTER } }
I20260501 08:11:23.436532 1662 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [sys.catalog]: This master's current role is: LEADER
I20260501 08:11:23.436708 1663 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [sys.catalog]: This master's current role is: LEADER
I20260501 08:11:23.436857 1670 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 08:11:23.437170 1670 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 08:11:23.437616 1100 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260501 08:11:23.437876 1670 catalog_manager.cc:1357] Generated new cluster ID: 659148a0b34f4bc397581b614c22c351
I20260501 08:11:23.437927 1670 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 08:11:23.446521 1670 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 08:11:23.446992 1670 catalog_manager.cc:1514] Loading token signing keys...
I20260501 08:11:23.455703 1670 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1: Generated new TSK 0
I20260501 08:11:23.455842 1670 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 08:11:23.469851 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260501 08:11:23.470034 1100 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260501 08:11:23.471617 1679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:23.471693 1682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:23.471776 1100 server_base.cc:1061] running on GCE node
W20260501 08:11:23.471967 1680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:23.472118 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:23.472200 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:23.472222 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623083472223 us; error 0 us; skew 500 ppm
I20260501 08:11:23.472813 1100 webserver.cc:492] Webserver started at http://127.1.19.1:43661/ using document root <none> and password file <none>
I20260501 08:11:23.472889 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:23.472930 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:23.472960 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:23.473215 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-0-root/instance:
uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
format_stamp: "Formatted at 2026-05-01 08:11:23 on dist-test-slave-pv7w"
I20260501 08:11:23.474200 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:23.474653 1687 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.474864 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.474947 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-0-root
uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
format_stamp: "Formatted at 2026-05-01 08:11:23 on dist-test-slave-pv7w"
I20260501 08:11:23.475029 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:23.490861 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:23.491114 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:23.491304 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:23.491539 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:11:23.491586 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.491629 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:11:23.491657 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.495016 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.1:39211
I20260501 08:11:23.495047 1750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.1:39211 every 8 connection(s)
I20260501 08:11:23.495452 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260501 08:11:23.495585 1100 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260501 08:11:23.496876 1755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:23.496886 1756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:23.497045 1758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:23.497225 1100 server_base.cc:1061] running on GCE node
I20260501 08:11:23.497363 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:23.497493 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:23.497545 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623083497544 us; error 0 us; skew 500 ppm
I20260501 08:11:23.498246 1100 webserver.cc:492] Webserver started at http://127.1.19.2:34675/ using document root <none> and password file <none>
I20260501 08:11:23.498351 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:23.498400 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:23.498452 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:23.498824 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-1-root/instance:
uuid: "a0a0ccfc23374b689a7cebe47834418b"
format_stamp: "Formatted at 2026-05-01 08:11:23 on dist-test-slave-pv7w"
I20260501 08:11:23.499589 1751 heartbeater.cc:344] Connected to a master server at 127.1.19.62:39201
I20260501 08:11:23.499665 1751 heartbeater.cc:461] Registering TS with master...
I20260501 08:11:23.499794 1751 heartbeater.cc:507] Master 127.1.19.62:39201 requested a full tablet report, sending...
I20260501 08:11:23.500072 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:23.500075 1622 ts_manager.cc:194] Registered new tserver with Master: 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211)
I20260501 08:11:23.500746 1763 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.500887 1622 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53440
I20260501 08:11:23.500908 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260501 08:11:23.500965 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-1-root
uuid: "a0a0ccfc23374b689a7cebe47834418b"
format_stamp: "Formatted at 2026-05-01 08:11:23 on dist-test-slave-pv7w"
I20260501 08:11:23.501014 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:23.511900 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:23.512101 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:23.512332 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:23.512557 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:11:23.512586 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.512609 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:11:23.512651 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.515625 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.2:46551
I20260501 08:11:23.515712 1826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.2:46551 every 8 connection(s)
I20260501 08:11:23.516052 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260501 08:11:23.516245 1100 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260501 08:11:23.517685 1831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:23.517695 1830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:23.517761 1833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:23.518021 1100 server_base.cc:1061] running on GCE node
I20260501 08:11:23.518147 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:23.518178 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:23.518195 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623083518196 us; error 0 us; skew 500 ppm
I20260501 08:11:23.518821 1100 webserver.cc:492] Webserver started at http://127.1.19.3:43809/ using document root <none> and password file <none>
I20260501 08:11:23.518906 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:23.518970 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:23.519055 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:23.519490 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-2-root/instance:
uuid: "5cedf472a6394d4296c037d18340cb1e"
format_stamp: "Formatted at 2026-05-01 08:11:23 on dist-test-slave-pv7w"
I20260501 08:11:23.519794 1827 heartbeater.cc:344] Connected to a master server at 127.1.19.62:39201
I20260501 08:11:23.519883 1827 heartbeater.cc:461] Registering TS with master...
I20260501 08:11:23.520162 1827 heartbeater.cc:507] Master 127.1.19.62:39201 requested a full tablet report, sending...
I20260501 08:11:23.520454 1622 ts_manager.cc:194] Registered new tserver with Master: a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:23.520792 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:23.520946 1622 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53444
I20260501 08:11:23.521373 1838 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.521507 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260501 08:11:23.521577 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-2-root
uuid: "5cedf472a6394d4296c037d18340cb1e"
format_stamp: "Formatted at 2026-05-01 08:11:23 on dist-test-slave-pv7w"
I20260501 08:11:23.521627 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:23.529876 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:23.530167 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:23.530359 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:23.530571 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:11:23.530620 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.530663 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:11:23.530690 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.533953 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.3:37871
I20260501 08:11:23.534052 1901 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.3:37871 every 8 connection(s)
I20260501 08:11:23.537459 1902 heartbeater.cc:344] Connected to a master server at 127.1.19.62:39201
I20260501 08:11:23.537549 1902 heartbeater.cc:461] Registering TS with master...
I20260501 08:11:23.537680 1902 heartbeater.cc:507] Master 127.1.19.62:39201 requested a full tablet report, sending...
I20260501 08:11:23.537909 1622 ts_manager.cc:194] Registered new tserver with Master: 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871)
I20260501 08:11:23.538257 1622 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53452
I20260501 08:11:23.538838 1100 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004648368s
I20260501 08:11:23.551994 1622 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53482:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 08:11:23.552199 1622 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 08:11:23.557238 1715 tablet_service.cc:1511] Processing CreateTablet for tablet b075b51cc8d04cc5bb00758973fbb343 (DEFAULT_TABLE table=test-workload [id=0360018bb0d547d8abcd692a100c4c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 08:11:23.557418 1715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b075b51cc8d04cc5bb00758973fbb343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:11:23.557495 1866 tablet_service.cc:1511] Processing CreateTablet for tablet b075b51cc8d04cc5bb00758973fbb343 (DEFAULT_TABLE table=test-workload [id=0360018bb0d547d8abcd692a100c4c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 08:11:23.557576 1866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b075b51cc8d04cc5bb00758973fbb343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:11:23.557577 1791 tablet_service.cc:1511] Processing CreateTablet for tablet b075b51cc8d04cc5bb00758973fbb343 (DEFAULT_TABLE table=test-workload [id=0360018bb0d547d8abcd692a100c4c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 08:11:23.557760 1791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b075b51cc8d04cc5bb00758973fbb343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:11:23.558969 1924 tablet_bootstrap.cc:492] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e: Bootstrap starting.
I20260501 08:11:23.559520 1924 tablet_bootstrap.cc:654] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e: Neither blocks nor log segments found. Creating new log.
I20260501 08:11:23.559821 1925 tablet_bootstrap.cc:492] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: Bootstrap starting.
I20260501 08:11:23.560276 1925 tablet_bootstrap.cc:654] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: Neither blocks nor log segments found. Creating new log.
I20260501 08:11:23.560282 1924 tablet_bootstrap.cc:492] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e: No bootstrap required, opened a new log
I20260501 08:11:23.560384 1924 ts_tablet_manager.cc:1403] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:23.560676 1924 raft_consensus.cc:359] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.560783 1923 tablet_bootstrap.cc:492] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: Bootstrap starting.
I20260501 08:11:23.560782 1924 raft_consensus.cc:385] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:11:23.560891 1924 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cedf472a6394d4296c037d18340cb1e, State: Initialized, Role: FOLLOWER
I20260501 08:11:23.560950 1924 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.561143 1924 ts_tablet_manager.cc:1434] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260501 08:11:23.561195 1902 heartbeater.cc:499] Master 127.1.19.62:39201 was elected leader, sending a full tablet report...
I20260501 08:11:23.561254 1923 tablet_bootstrap.cc:654] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: Neither blocks nor log segments found. Creating new log.
I20260501 08:11:23.561733 1925 tablet_bootstrap.cc:492] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: No bootstrap required, opened a new log
I20260501 08:11:23.561787 1925 ts_tablet_manager.cc:1403] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260501 08:11:23.561893 1923 tablet_bootstrap.cc:492] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: No bootstrap required, opened a new log
I20260501 08:11:23.561945 1925 raft_consensus.cc:359] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.562044 1925 raft_consensus.cc:385] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:11:23.562000 1923 ts_tablet_manager.cc:1403] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:23.562088 1925 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254a82ec6f2e495cbfb6d4f83d19a0f6, State: Initialized, Role: FOLLOWER
I20260501 08:11:23.562153 1925 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.562223 1923 raft_consensus.cc:359] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.562297 1923 raft_consensus.cc:385] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:11:23.562321 1923 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a0ccfc23374b689a7cebe47834418b, State: Initialized, Role: FOLLOWER
I20260501 08:11:23.562355 1751 heartbeater.cc:499] Master 127.1.19.62:39201 was elected leader, sending a full tablet report...
I20260501 08:11:23.562400 1923 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.562328 1925 ts_tablet_manager.cc:1434] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260501 08:11:23.562578 1923 ts_tablet_manager.cc:1434] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:23.562696 1827 heartbeater.cc:499] Master 127.1.19.62:39201 was elected leader, sending a full tablet report...
I20260501 08:11:23.586229 1928 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 08:11:23.586354 1928 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.586625 1928 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:23.589681 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" is_pre_election: true
I20260501 08:11:23.589777 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b" is_pre_election: true
I20260501 08:11:23.589828 1725 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cedf472a6394d4296c037d18340cb1e in term 0.
I20260501 08:11:23.589875 1801 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cedf472a6394d4296c037d18340cb1e in term 0.
I20260501 08:11:23.590085 1839 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cedf472a6394d4296c037d18340cb1e, a0a0ccfc23374b689a7cebe47834418b; no voters:
I20260501 08:11:23.590226 1928 raft_consensus.cc:2804] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 08:11:23.590283 1928 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:23.590312 1928 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:11:23.590996 1928 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.591104 1928 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 1 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:23.591300 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:23.591339 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:23.591367 1725 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:11:23.591378 1801 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:11:23.592064 1725 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cedf472a6394d4296c037d18340cb1e in term 1.
I20260501 08:11:23.592064 1801 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cedf472a6394d4296c037d18340cb1e in term 1.
I20260501 08:11:23.592281 1839 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cedf472a6394d4296c037d18340cb1e, a0a0ccfc23374b689a7cebe47834418b; no voters:
I20260501 08:11:23.592381 1928 raft_consensus.cc:2804] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 1 FOLLOWER]: Leader election won for term 1
I20260501 08:11:23.592545 1928 raft_consensus.cc:697] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 1 LEADER]: Becoming Leader. State: Replica: 5cedf472a6394d4296c037d18340cb1e, State: Running, Role: LEADER
I20260501 08:11:23.592588 1928 consensus_queue.cc:237] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.593267 1621 catalog_manager.cc:5671] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cedf472a6394d4296c037d18340cb1e (127.1.19.3). New cstate: current_term: 1 leader_uuid: "5cedf472a6394d4296c037d18340cb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:23.606751 1801 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 1 FOLLOWER]: Refusing update from remote peer 5cedf472a6394d4296c037d18340cb1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 08:11:23.606828 1725 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 1 FOLLOWER]: Refusing update from remote peer 5cedf472a6394d4296c037d18340cb1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 08:11:23.607118 1933 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 08:11:23.607277 1928 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 08:11:23.828879 1930 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5cedf472a6394d4296c037d18340cb1e)
I20260501 08:11:23.829034 1930 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 1 FOLLOWER]: Advancing to term 2
I20260501 08:11:23.829550 1931 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5cedf472a6394d4296c037d18340cb1e)
I20260501 08:11:23.829618 1931 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 1 FOLLOWER]: Advancing to term 2
I20260501 08:11:23.830050 1930 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.830294 1930 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 2 election: Requested vote from peers 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:23.830370 1931 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.830542 1931 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 2 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871)
I20260501 08:11:23.833912 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:23.833961 1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:23.834049 1801 raft_consensus.cc:2393] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 254a82ec6f2e495cbfb6d4f83d19a0f6 in current term 2: Already voted for candidate a0a0ccfc23374b689a7cebe47834418b in this term.
I20260501 08:11:23.834143 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:23.834249 1689 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 254a82ec6f2e495cbfb6d4f83d19a0f6; no voters: 5cedf472a6394d4296c037d18340cb1e, a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:23.834363 1930 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260501 08:11:23.834451 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:23.834512 1725 raft_consensus.cc:2393] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a0a0ccfc23374b689a7cebe47834418b in current term 2: Already voted for candidate 254a82ec6f2e495cbfb6d4f83d19a0f6 in this term.
I20260501 08:11:23.834725 1764 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0a0ccfc23374b689a7cebe47834418b; no voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, 5cedf472a6394d4296c037d18340cb1e
I20260501 08:11:23.834895 1931 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260501 08:11:23.835397 1725 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 2 FOLLOWER]: Rejecting Update request from peer 5cedf472a6394d4296c037d18340cb1e for earlier term 1. Current term is 2. Ops: []
I20260501 08:11:23.835656 1945 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 }, Status: INVALID_TERM, Last received: 1.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260501 08:11:23.843717 1801 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 2 FOLLOWER]: Rejecting Update request from peer 5cedf472a6394d4296c037d18340cb1e for earlier term 1. Current term is 2. Ops: []
I20260501 08:11:23.844094 1928 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 }, Status: INVALID_TERM, Last received: 1.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260501 08:11:23.862869 1945 raft_consensus.cc:3055] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 1 LEADER]: Stepping down as leader of term 1
I20260501 08:11:23.862985 1945 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5cedf472a6394d4296c037d18340cb1e, State: Running, Role: LEADER
I20260501 08:11:23.863104 1945 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 18, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.863226 1945 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 1 FOLLOWER]: Advancing to term 2
I20260501 08:11:23.961874 1930 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:23.962008 1930 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 2 FOLLOWER]: Advancing to term 3
I20260501 08:11:23.963122 1930 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.963270 1931 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:23.963362 1931 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 2 FOLLOWER]: Advancing to term 3
I20260501 08:11:23.963471 1930 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 3 election: Requested vote from peers 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:23.963655 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 3 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:23.963666 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 3 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:23.963949 1875 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 2 FOLLOWER]: Advancing to term 3
I20260501 08:11:23.964320 1931 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.964445 1801 raft_consensus.cc:2393] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 254a82ec6f2e495cbfb6d4f83d19a0f6 in current term 3: Already voted for candidate a0a0ccfc23374b689a7cebe47834418b in this term.
I20260501 08:11:23.964493 1931 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 3 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871)
I20260501 08:11:23.964775 1875 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 254a82ec6f2e495cbfb6d4f83d19a0f6 in term 3.
I20260501 08:11:23.964811 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 3 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:23.964869 1725 raft_consensus.cc:2393] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0a0ccfc23374b689a7cebe47834418b in current term 3: Already voted for candidate 254a82ec6f2e495cbfb6d4f83d19a0f6 in this term.
I20260501 08:11:23.965003 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 3 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:23.965022 1689 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, 5cedf472a6394d4296c037d18340cb1e; no voters: a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:23.965095 1875 raft_consensus.cc:2393] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a0a0ccfc23374b689a7cebe47834418b in current term 3: Already voted for candidate 254a82ec6f2e495cbfb6d4f83d19a0f6 in this term.
I20260501 08:11:23.965133 1930 raft_consensus.cc:2804] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 3 FOLLOWER]: Leader election won for term 3
I20260501 08:11:23.965256 1930 raft_consensus.cc:697] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 3 LEADER]: Becoming Leader. State: Replica: 254a82ec6f2e495cbfb6d4f83d19a0f6, State: Running, Role: LEADER
I20260501 08:11:23.965340 1930 consensus_queue.cc:237] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 18, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:23.965214 1765 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0a0ccfc23374b689a7cebe47834418b; no voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, 5cedf472a6394d4296c037d18340cb1e
I20260501 08:11:23.965524 1931 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260501 08:11:23.965917 1621 catalog_manager.cc:5671] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 reported cstate change: term changed from 1 to 3, leader changed from 5cedf472a6394d4296c037d18340cb1e (127.1.19.3) to 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1). New cstate: current_term: 3 leader_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:24.043403 1801 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 3 FOLLOWER]: Refusing update from remote peer 254a82ec6f2e495cbfb6d4f83d19a0f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20260501 08:11:24.043737 1930 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260501 08:11:24.082671 1875 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 3 FOLLOWER]: Refusing update from remote peer 254a82ec6f2e495cbfb6d4f83d19a0f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20260501 08:11:24.083016 1930 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260501 08:11:24.183885 1928 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 254a82ec6f2e495cbfb6d4f83d19a0f6)
I20260501 08:11:24.184015 1928 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 3 FOLLOWER]: Advancing to term 4
I20260501 08:11:24.184686 1928 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.184866 1928 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 4 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:24.185123 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:24.185175 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:24.185411 1839 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cedf472a6394d4296c037d18340cb1e; no voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:24.185527 1928 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260501 08:11:24.206048 1875 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 4 FOLLOWER]: Rejecting Update request from peer 254a82ec6f2e495cbfb6d4f83d19a0f6 for earlier term 3. Current term is 4. Ops: []
I20260501 08:11:24.206439 1962 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 }, Status: INVALID_TERM, Last received: 3.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260501 08:11:24.289762 1928 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:24.289921 1928 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 4 FOLLOWER]: Advancing to term 5
I20260501 08:11:24.290617 1928 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.290799 1928 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 5 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:24.291122 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 5 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:24.291132 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 5 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:24.291420 1839 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cedf472a6394d4296c037d18340cb1e; no voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:24.291527 1928 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260501 08:11:24.300724 1962 raft_consensus.cc:3055] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 3 LEADER]: Stepping down as leader of term 3
I20260501 08:11:24.300823 1962 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 254a82ec6f2e495cbfb6d4f83d19a0f6, State: Running, Role: LEADER
I20260501 08:11:24.300902 1962 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 3.19, Last appended by leader: 19, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.301020 1962 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 3 FOLLOWER]: Advancing to term 4
I20260501 08:11:24.301050 1875 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 5 FOLLOWER]: Rejecting Update request from peer 254a82ec6f2e495cbfb6d4f83d19a0f6 for earlier term 3. Current term is 5. Ops: []
I20260501 08:11:24.336781 1931 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 254a82ec6f2e495cbfb6d4f83d19a0f6)
I20260501 08:11:24.336871 1931 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 3 FOLLOWER]: Advancing to term 4
I20260501 08:11:24.337623 1931 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.337805 1931 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 4 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871)
I20260501 08:11:24.338148 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:24.338148 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:24.338346 1875 raft_consensus.cc:2368] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0a0ccfc23374b689a7cebe47834418b for earlier term 4. Current term is 5.
I20260501 08:11:24.338526 1765 leader_election.cc:400] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 4 election: Vote denied by peer 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871) with higher term. Message: Invalid argument: T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0a0ccfc23374b689a7cebe47834418b for earlier term 4. Current term is 5.
I20260501 08:11:24.338564 1765 leader_election.cc:403] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260501 08:11:24.338621 1931 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 4 FOLLOWER]: Advancing to term 5
I20260501 08:11:24.339118 1725 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a0ccfc23374b689a7cebe47834418b in term 4.
I20260501 08:11:24.339339 1931 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871) with higher term. Message: Invalid argument: T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a0a0ccfc23374b689a7cebe47834418b for earlier term 4. Current term is 5.
I20260501 08:11:24.432153 1928 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:24.432276 1928 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 5 FOLLOWER]: Advancing to term 6
I20260501 08:11:24.432938 1928 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.433130 1928 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 6 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:24.433372 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 6 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:24.433444 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 6 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:24.433465 1725 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 4 FOLLOWER]: Advancing to term 6
I20260501 08:11:24.433597 1801 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 5 FOLLOWER]: Advancing to term 6
I20260501 08:11:24.434262 1725 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cedf472a6394d4296c037d18340cb1e in term 6.
I20260501 08:11:24.434389 1801 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cedf472a6394d4296c037d18340cb1e in term 6.
I20260501 08:11:24.434599 1839 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cedf472a6394d4296c037d18340cb1e, a0a0ccfc23374b689a7cebe47834418b; no voters:
I20260501 08:11:24.434724 1928 raft_consensus.cc:2804] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 6 FOLLOWER]: Leader election won for term 6
I20260501 08:11:24.434791 1928 raft_consensus.cc:697] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 6 LEADER]: Becoming Leader. State: Replica: 5cedf472a6394d4296c037d18340cb1e, State: Running, Role: LEADER
I20260501 08:11:24.434850 1928 consensus_queue.cc:237] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 3.19, Last appended by leader: 19, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.435390 1621 catalog_manager.cc:5671] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e reported cstate change: term changed from 3 to 6, leader changed from 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1) to 5cedf472a6394d4296c037d18340cb1e (127.1.19.3). New cstate: current_term: 6 leader_uuid: "5cedf472a6394d4296c037d18340cb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:24.514847 1725 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 6 FOLLOWER]: Refusing update from remote peer 5cedf472a6394d4296c037d18340cb1e: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 6 index: 20. (index mismatch)
I20260501 08:11:24.515262 1928 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260501 08:11:24.536062 1801 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 6 FOLLOWER]: Refusing update from remote peer 5cedf472a6394d4296c037d18340cb1e: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 6 index: 20. (index mismatch)
I20260501 08:11:24.536478 1928 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260501 08:11:24.617343 1962 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5cedf472a6394d4296c037d18340cb1e)
I20260501 08:11:24.617494 1962 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 6 FOLLOWER]: Advancing to term 7
I20260501 08:11:24.618279 1962 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.618505 1962 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 7 election: Requested vote from peers 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:24.618743 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:24.618851 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:24.619201 1688 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 254a82ec6f2e495cbfb6d4f83d19a0f6; no voters: 5cedf472a6394d4296c037d18340cb1e, a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:24.619319 1962 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260501 08:11:24.619379 1725 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 7 FOLLOWER]: Rejecting Update request from peer 5cedf472a6394d4296c037d18340cb1e for earlier term 6. Current term is 7. Ops: []
I20260501 08:11:24.619606 1945 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 }, Status: INVALID_TERM, Last received: 6.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260501 08:11:24.622831 1945 raft_consensus.cc:3055] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 6 LEADER]: Stepping down as leader of term 6
I20260501 08:11:24.622895 1945 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5cedf472a6394d4296c037d18340cb1e, State: Running, Role: LEADER
I20260501 08:11:24.622952 1945 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.623070 1945 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 6 FOLLOWER]: Advancing to term 7
I20260501 08:11:24.722491 1931 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5cedf472a6394d4296c037d18340cb1e)
I20260501 08:11:24.722610 1931 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 6 FOLLOWER]: Advancing to term 7
I20260501 08:11:24.723397 1931 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.723613 1931 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 7 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871)
I20260501 08:11:24.723855 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:24.723868 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:24.724018 1725 raft_consensus.cc:2393] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a0a0ccfc23374b689a7cebe47834418b in current term 7: Already voted for candidate 254a82ec6f2e495cbfb6d4f83d19a0f6 in this term.
I20260501 08:11:24.724624 1875 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a0ccfc23374b689a7cebe47834418b in term 7.
I20260501 08:11:24.724860 1765 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cedf472a6394d4296c037d18340cb1e, a0a0ccfc23374b689a7cebe47834418b; no voters: 254a82ec6f2e495cbfb6d4f83d19a0f6
I20260501 08:11:24.725055 1931 raft_consensus.cc:2804] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 7 FOLLOWER]: Leader election won for term 7
I20260501 08:11:24.725179 1931 raft_consensus.cc:697] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 7 LEADER]: Becoming Leader. State: Replica: a0a0ccfc23374b689a7cebe47834418b, State: Running, Role: LEADER
I20260501 08:11:24.725219 1931 consensus_queue.cc:237] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.725781 1620 catalog_manager.cc:5671] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b reported cstate change: term changed from 6 to 7, leader changed from 5cedf472a6394d4296c037d18340cb1e (127.1.19.3) to a0a0ccfc23374b689a7cebe47834418b (127.1.19.2). New cstate: current_term: 7 leader_uuid: "a0a0ccfc23374b689a7cebe47834418b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:24.739454 1962 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:24.739609 1962 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 7 FOLLOWER]: Advancing to term 8
I20260501 08:11:24.740465 1962 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.740684 1962 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 8 election: Requested vote from peers 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:24.740918 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 8 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:24.740967 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 8 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:24.741060 1875 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 7 FOLLOWER]: Advancing to term 8
I20260501 08:11:24.741806 1875 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 254a82ec6f2e495cbfb6d4f83d19a0f6 in term 8.
I20260501 08:11:24.742017 1689 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, 5cedf472a6394d4296c037d18340cb1e; no voters: a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:24.742117 1962 raft_consensus.cc:2804] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 8 FOLLOWER]: Leader election won for term 8
I20260501 08:11:24.742193 1962 raft_consensus.cc:697] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 8 LEADER]: Becoming Leader. State: Replica: 254a82ec6f2e495cbfb6d4f83d19a0f6, State: Running, Role: LEADER
I20260501 08:11:24.742247 1962 consensus_queue.cc:237] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.781908 1621 catalog_manager.cc:5671] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 reported cstate change: term changed from 7 to 8, leader changed from a0a0ccfc23374b689a7cebe47834418b (127.1.19.2) to 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1). New cstate: current_term: 8 leader_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:24.811375 1875 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 8 FOLLOWER]: Rejecting Update request from peer a0a0ccfc23374b689a7cebe47834418b for earlier term 7. Current term is 8. Ops: []
I20260501 08:11:24.811736 1931 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260501 08:11:24.829001 1931 raft_consensus.cc:3055] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 7 LEADER]: Stepping down as leader of term 7
I20260501 08:11:24.829124 1931 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a0a0ccfc23374b689a7cebe47834418b, State: Running, Role: LEADER
I20260501 08:11:24.829233 1931 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 7.21, Last appended by leader: 21, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:24.829377 1931 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 7 FOLLOWER]: Advancing to term 8
I20260501 08:11:24.853955 1801 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 8 FOLLOWER]: Refusing update from remote peer 254a82ec6f2e495cbfb6d4f83d19a0f6: Log matching property violated. Preceding OpId in replica: term: 7 index: 21. Preceding OpId from leader: term: 8 index: 21. (term mismatch)
I20260501 08:11:24.854064 1801 pending_rounds.cc:85] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: Aborting all ops after (but not including) 20
I20260501 08:11:24.854084 1801 pending_rounds.cc:107] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: Aborting uncommitted NO_OP operation due to leader change: 7.21
I20260501 08:11:24.854107 1801 raft_consensus.cc:2889] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:24.854355 1962 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260501 08:11:24.857898 1875 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 8 FOLLOWER]: Refusing update from remote peer 254a82ec6f2e495cbfb6d4f83d19a0f6: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 8 index: 21. (index mismatch)
I20260501 08:11:24.858232 1962 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260501 08:11:25.058166 1945 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 254a82ec6f2e495cbfb6d4f83d19a0f6)
I20260501 08:11:25.058275 1945 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 8 FOLLOWER]: Advancing to term 9
I20260501 08:11:25.058992 1945 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.059165 1945 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 9 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:25.059518 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 9 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:25.059580 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 9 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:25.059836 1839 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cedf472a6394d4296c037d18340cb1e; no voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:25.059975 1945 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260501 08:11:25.066617 1875 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 9 FOLLOWER]: Rejecting Update request from peer 254a82ec6f2e495cbfb6d4f83d19a0f6 for earlier term 8. Current term is 9. Ops: []
I20260501 08:11:25.066989 1965 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 }, Status: INVALID_TERM, Last received: 8.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260501 08:11:25.136274 1965 raft_consensus.cc:3055] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 8 LEADER]: Stepping down as leader of term 8
I20260501 08:11:25.136415 1965 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 254a82ec6f2e495cbfb6d4f83d19a0f6, State: Running, Role: LEADER
I20260501 08:11:25.136548 1965 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 8.21, Last appended by leader: 21, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.136708 1965 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 8 FOLLOWER]: Advancing to term 9
I20260501 08:11:25.160490 1931 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 254a82ec6f2e495cbfb6d4f83d19a0f6)
I20260501 08:11:25.160595 1931 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 8 FOLLOWER]: Advancing to term 9
I20260501 08:11:25.161340 1931 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.161540 1931 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 9 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871)
I20260501 08:11:25.161803 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 9 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:25.161803 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 9 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:25.161952 1875 raft_consensus.cc:2393] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a0a0ccfc23374b689a7cebe47834418b in current term 9: Already voted for candidate 5cedf472a6394d4296c037d18340cb1e in this term.
I20260501 08:11:25.162578 1725 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a0ccfc23374b689a7cebe47834418b in term 9.
I20260501 08:11:25.162763 1764 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, a0a0ccfc23374b689a7cebe47834418b; no voters: 5cedf472a6394d4296c037d18340cb1e
I20260501 08:11:25.162864 1931 raft_consensus.cc:2804] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 9 FOLLOWER]: Leader election won for term 9
I20260501 08:11:25.162951 1931 raft_consensus.cc:697] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 9 LEADER]: Becoming Leader. State: Replica: a0a0ccfc23374b689a7cebe47834418b, State: Running, Role: LEADER
I20260501 08:11:25.162999 1931 consensus_queue.cc:237] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 8.21, Last appended by leader: 21, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.163493 1621 catalog_manager.cc:5671] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b reported cstate change: term changed from 8 to 9, leader changed from 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1) to a0a0ccfc23374b689a7cebe47834418b (127.1.19.2). New cstate: current_term: 9 leader_uuid: "a0a0ccfc23374b689a7cebe47834418b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:25.205437 1945 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:25.205542 1945 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 9 FOLLOWER]: Advancing to term 10
I20260501 08:11:25.206249 1945 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.206386 1945 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 10 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:25.206673 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 10 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:25.206748 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 10 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:25.206765 1725 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 9 FOLLOWER]: Advancing to term 10
I20260501 08:11:25.207463 1725 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cedf472a6394d4296c037d18340cb1e in term 10.
I20260501 08:11:25.207670 1839 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, 5cedf472a6394d4296c037d18340cb1e; no voters: a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:25.207791 1945 raft_consensus.cc:2804] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 10 FOLLOWER]: Leader election won for term 10
I20260501 08:11:25.207880 1945 raft_consensus.cc:697] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 10 LEADER]: Becoming Leader. State: Replica: 5cedf472a6394d4296c037d18340cb1e, State: Running, Role: LEADER
I20260501 08:11:25.207934 1945 consensus_queue.cc:237] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 8.21, Last appended by leader: 21, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.256487 1620 catalog_manager.cc:5671] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e reported cstate change: term changed from 9 to 10, leader changed from a0a0ccfc23374b689a7cebe47834418b (127.1.19.2) to 5cedf472a6394d4296c037d18340cb1e (127.1.19.3). New cstate: current_term: 10 leader_uuid: "5cedf472a6394d4296c037d18340cb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:25.267462 1725 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 10 FOLLOWER]: Rejecting Update request from peer a0a0ccfc23374b689a7cebe47834418b for earlier term 9. Current term is 10. Ops: []
I20260501 08:11:25.267858 1931 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260501 08:11:25.268878 1875 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 10 LEADER]: Rejecting Update request from peer a0a0ccfc23374b689a7cebe47834418b for earlier term 9. Current term is 10. Ops: []
I20260501 08:11:25.269359 1964 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260501 08:11:25.269447 1964 consensus_queue.cc:1243] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [LEADER]: Peer 5cedf472a6394d4296c037d18340cb1e log is divergent from this leader: its last log entry 10.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260501 08:11:25.283883 1725 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 10 FOLLOWER]: Refusing update from remote peer 5cedf472a6394d4296c037d18340cb1e: Log matching property violated. Preceding OpId in replica: term: 8 index: 21. Preceding OpId from leader: term: 10 index: 22. (index mismatch)
I20260501 08:11:25.284330 1945 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260501 08:11:25.292503 1801 raft_consensus.cc:3055] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 9 LEADER]: Stepping down as leader of term 9
I20260501 08:11:25.292605 1801 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a0a0ccfc23374b689a7cebe47834418b, State: Running, Role: LEADER
I20260501 08:11:25.292678 1801 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 9.22, Last appended by leader: 22, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.292801 1801 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 9 FOLLOWER]: Advancing to term 10
I20260501 08:11:25.293563 1801 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 10 FOLLOWER]: Refusing update from remote peer 5cedf472a6394d4296c037d18340cb1e: Log matching property violated. Preceding OpId in replica: term: 9 index: 22. Preceding OpId from leader: term: 10 index: 22. (term mismatch)
I20260501 08:11:25.293649 1801 pending_rounds.cc:85] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: Aborting all ops after (but not including) 21
I20260501 08:11:25.293673 1801 pending_rounds.cc:107] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: Aborting uncommitted NO_OP operation due to leader change: 9.22
I20260501 08:11:25.293705 1801 raft_consensus.cc:2889] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:25.294111 1945 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260501 08:11:25.386516 1965 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5cedf472a6394d4296c037d18340cb1e)
I20260501 08:11:25.386623 1965 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 10 FOLLOWER]: Advancing to term 11
I20260501 08:11:25.387341 1965 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.387538 1965 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 11 election: Requested vote from peers 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:25.387768 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 11 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:25.387809 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 11 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:25.388150 1688 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 254a82ec6f2e495cbfb6d4f83d19a0f6; no voters: 5cedf472a6394d4296c037d18340cb1e, a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:25.388267 1965 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260501 08:11:25.391479 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:25.392735 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:25.395062 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:25.399333 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:25.406450 1725 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 11 FOLLOWER]: Rejecting Update request from peer 5cedf472a6394d4296c037d18340cb1e for earlier term 10. Current term is 11. Ops: []
I20260501 08:11:25.406827 1928 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 }, Status: INVALID_TERM, Last received: 10.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260501 08:11:25.407577 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:25.423830 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:25.456243 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:25.483994 1725 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 11 FOLLOWER]: Rejecting Update request from peer 5cedf472a6394d4296c037d18340cb1e for earlier term 10. Current term is 11. Ops: []
I20260501 08:11:25.484398 1928 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 }, Status: INVALID_TERM, Last received: 10.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260501 08:11:25.487210 1966 raft_consensus.cc:3055] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 10 LEADER]: Stepping down as leader of term 10
I20260501 08:11:25.487313 1966 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5cedf472a6394d4296c037d18340cb1e, State: Running, Role: LEADER
I20260501 08:11:25.487396 1966 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 10.22, Last appended by leader: 22, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.487524 1966 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 10 FOLLOWER]: Advancing to term 11
I20260501 08:11:25.518565 1931 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5cedf472a6394d4296c037d18340cb1e)
I20260501 08:11:25.518673 1931 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 10 FOLLOWER]: Advancing to term 11
I20260501 08:11:25.519491 1931 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.519696 1931 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 11 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871)
I20260501 08:11:25.519955 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 11 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:25.520051 1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "a0a0ccfc23374b689a7cebe47834418b" candidate_term: 11 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
I20260501 08:11:25.520259 1725 raft_consensus.cc:2393] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a0a0ccfc23374b689a7cebe47834418b in current term 11: Already voted for candidate 254a82ec6f2e495cbfb6d4f83d19a0f6 in this term.
I20260501 08:11:25.520517 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:25.521071 1875 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a0ccfc23374b689a7cebe47834418b in term 11.
I20260501 08:11:25.521313 1765 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cedf472a6394d4296c037d18340cb1e, a0a0ccfc23374b689a7cebe47834418b; no voters: 254a82ec6f2e495cbfb6d4f83d19a0f6
I20260501 08:11:25.521420 1931 raft_consensus.cc:2804] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 11 FOLLOWER]: Leader election won for term 11
I20260501 08:11:25.521512 1931 raft_consensus.cc:697] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 11 LEADER]: Becoming Leader. State: Replica: a0a0ccfc23374b689a7cebe47834418b, State: Running, Role: LEADER
I20260501 08:11:25.521590 1931 consensus_queue.cc:237] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 10.22, Last appended by leader: 22, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.522137 1622 catalog_manager.cc:5671] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b reported cstate change: term changed from 10 to 11, leader changed from 5cedf472a6394d4296c037d18340cb1e (127.1.19.3) to a0a0ccfc23374b689a7cebe47834418b (127.1.19.2). New cstate: current_term: 11 leader_uuid: "a0a0ccfc23374b689a7cebe47834418b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:25.524794 1965 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:25.524860 1965 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 11 FOLLOWER]: Advancing to term 12
I20260501 08:11:25.525496 1965 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.525661 1965 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 12 election: Requested vote from peers 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:25.525836 1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 12 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "5cedf472a6394d4296c037d18340cb1e"
I20260501 08:11:25.525888 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" candidate_term: 12 candidate_status { last_received { term: 10 index: 22 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
I20260501 08:11:25.525929 1875 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 11 FOLLOWER]: Advancing to term 12
I20260501 08:11:25.526564 1875 raft_consensus.cc:2468] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 254a82ec6f2e495cbfb6d4f83d19a0f6 in term 12.
I20260501 08:11:25.526794 1689 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, 5cedf472a6394d4296c037d18340cb1e; no voters: a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:25.526892 1965 raft_consensus.cc:2804] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 12 FOLLOWER]: Leader election won for term 12
I20260501 08:11:25.526990 1965 raft_consensus.cc:697] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 12 LEADER]: Becoming Leader. State: Replica: 254a82ec6f2e495cbfb6d4f83d19a0f6, State: Running, Role: LEADER
I20260501 08:11:25.527045 1965 consensus_queue.cc:237] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 10.22, Last appended by leader: 22, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.527560 1622 catalog_manager.cc:5671] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 reported cstate change: term changed from 11 to 12, leader changed from a0a0ccfc23374b689a7cebe47834418b (127.1.19.2) to 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1). New cstate: current_term: 12 leader_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } health_report { overall_health: UNKNOWN } } }
I20260501 08:11:25.618266 1725 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 12 LEADER]: Rejecting Update request from peer a0a0ccfc23374b689a7cebe47834418b for earlier term 11. Current term is 12. Ops: []
I20260501 08:11:25.618587 1931 consensus_queue.cc:1059] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260501 08:11:25.618731 1931 consensus_queue.cc:1243] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [LEADER]: Peer 254a82ec6f2e495cbfb6d4f83d19a0f6 log is divergent from this leader: its last log entry 12.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260501 08:11:25.618763 1964 raft_consensus.cc:3055] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 11 LEADER]: Stepping down as leader of term 11
I20260501 08:11:25.618811 1964 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a0a0ccfc23374b689a7cebe47834418b, State: Running, Role: LEADER
I20260501 08:11:25.618899 1964 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 11.23, Last appended by leader: 23, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.619004 1964 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 11 FOLLOWER]: Advancing to term 12
I20260501 08:11:25.631481 1801 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 12 FOLLOWER]: Refusing update from remote peer 254a82ec6f2e495cbfb6d4f83d19a0f6: Log matching property violated. Preceding OpId in replica: term: 11 index: 23. Preceding OpId from leader: term: 12 index: 23. (term mismatch)
I20260501 08:11:25.631570 1801 pending_rounds.cc:85] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: Aborting all ops after (but not including) 22
I20260501 08:11:25.631584 1801 pending_rounds.cc:107] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: Aborting uncommitted NO_OP operation due to leader change: 11.23
I20260501 08:11:25.631600 1801 raft_consensus.cc:2889] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260501 08:11:25.632150 1965 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260501 08:11:25.633651 1875 raft_consensus.cc:1275] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 12 FOLLOWER]: Refusing update from remote peer 254a82ec6f2e495cbfb6d4f83d19a0f6: Log matching property violated. Preceding OpId in replica: term: 10 index: 22. Preceding OpId from leader: term: 12 index: 23. (index mismatch)
I20260501 08:11:25.633906 1967 consensus_queue.cc:1048] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260501 08:11:25.648772 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:25.648904 1100 log_reader.cc:385] T b075b51cc8d04cc5bb00758973fbb343: removed 1 log segments from log reader
I20260501 08:11:25.648999 1100 log.cc:1079] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: Deleting log segment in path: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-0-root/wals/b075b51cc8d04cc5bb00758973fbb343/wal-000000001 (ops 1-18)
I20260501 08:11:25.649493 1100 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20260501 08:11:25.649556 1100 master.cc:562] Master@127.1.19.62:39201 shutting down...
I20260501 08:11:25.678989 1100 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 08:11:25.679102 1100 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:25.679124 1100 tablet_replica.cc:333] T 00000000000000000000000000000000 P f3ec594bf2fe433884fb71a197c136a1: stopping tablet replica
I20260501 08:11:25.691562 1100 master.cc:584] Master@127.1.19.62:39201 shutdown complete.
I20260501 08:11:25.693039 1100 tablet_server.cc:179] TabletServer@127.1.19.1:0 shutting down...
I20260501 08:11:25.735319 1966 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 254a82ec6f2e495cbfb6d4f83d19a0f6)
I20260501 08:11:25.735423 1966 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 12 FOLLOWER]: Advancing to term 13
I20260501 08:11:25.736263 1966 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.736429 1966 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 13 election: Requested vote from peers 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
I20260501 08:11:25.736713 1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b075b51cc8d04cc5bb00758973fbb343" candidate_uuid: "5cedf472a6394d4296c037d18340cb1e" candidate_term: 13 candidate_status { last_received { term: 12 index: 23 } } ignore_live_leader: false dest_uuid: "a0a0ccfc23374b689a7cebe47834418b"
W20260501 08:11:25.737094 1839 leader_election.cc:336] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 254a82ec6f2e495cbfb6d4f83d19a0f6 (127.1.19.1:39211): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260501 08:11:25.737172 1839 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cedf472a6394d4296c037d18340cb1e; no voters: 254a82ec6f2e495cbfb6d4f83d19a0f6, a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:25.737294 1966 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260501 08:11:25.747007 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:25.747152 1100 tablet_replica.cc:333] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: stopping tablet replica
I20260501 08:11:25.747172 1875 raft_consensus.cc:1240] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 13 FOLLOWER]: Rejecting Update request from peer 254a82ec6f2e495cbfb6d4f83d19a0f6 for earlier term 12. Current term is 13. Ops: []
I20260501 08:11:25.747216 1100 raft_consensus.cc:2243] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 12 LEADER]: Raft consensus shutting down.
I20260501 08:11:25.747283 1100 raft_consensus.cc:2272] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 12 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:25.759613 1100 tablet_server.cc:196] TabletServer@127.1.19.1:0 shutdown complete.
I20260501 08:11:25.760746 1100 tablet_server.cc:179] TabletServer@127.1.19.2:0 shutting down...
I20260501 08:11:25.767669 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:25.767874 1100 tablet_replica.cc:333] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b: stopping tablet replica
I20260501 08:11:25.767944 1100 raft_consensus.cc:2243] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 12 FOLLOWER]: Raft consensus shutting down.
I20260501 08:11:25.768071 1100 raft_consensus.cc:2272] T b075b51cc8d04cc5bb00758973fbb343 P a0a0ccfc23374b689a7cebe47834418b [term 12 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:25.780602 1100 tablet_server.cc:196] TabletServer@127.1.19.2:0 shutdown complete.
I20260501 08:11:25.781555 1100 tablet_server.cc:179] TabletServer@127.1.19.3:0 shutting down...
I20260501 08:11:25.785831 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:25.786039 1100 tablet_replica.cc:333] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e: stopping tablet replica
I20260501 08:11:25.786106 1100 raft_consensus.cc:2243] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 13 FOLLOWER]: Raft consensus shutting down.
I20260501 08:11:25.786163 1100 raft_consensus.cc:2272] T b075b51cc8d04cc5bb00758973fbb343 P 5cedf472a6394d4296c037d18340cb1e [term 13 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:25.788180 1100 tablet_server.cc:196] TabletServer@127.1.19.3:0 shutdown complete.
I20260501 08:11:25.789012 1100 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20260501 08:11:25.789122 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260501 08:11:25.789209 1100 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260501 08:11:25.790665 1969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:25.790643 1972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:25.790701 1970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:25.790894 1100 server_base.cc:1061] running on GCE node
I20260501 08:11:25.790998 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:25.791043 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:25.791071 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623085791071 us; error 0 us; skew 500 ppm
I20260501 08:11:25.791675 1100 webserver.cc:492] Webserver started at http://127.1.19.1:43661/ using document root <none> and password file <none>
I20260501 08:11:25.791786 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:25.791837 1100 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260501 08:11:25.792644 1100 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:25.794222 1979 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:25.794391 1100 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:25.794447 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-0-root
uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6"
format_stamp: "Formatted at 2026-05-01 08:11:23 on dist-test-slave-pv7w"
I20260501 08:11:25.794500 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1777623050595939-1100-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5427
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:25.806957 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:25.807166 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:25.807322 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:25.807718 1986 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260501 08:11:25.808501 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260501 08:11:25.808563 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260501 08:11:25.808612 1100 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260501 08:11:25.809031 1100 ts_tablet_manager.cc:616] Registered 1 tablets
I20260501 08:11:25.809088 1986 tablet_bootstrap.cc:492] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: Bootstrap starting.
I20260501 08:11:25.809089 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:25.810499 1986 tablet_bootstrap.cc:492] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 08:11:25.810757 1986 tablet_bootstrap.cc:492] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: Bootstrap complete.
I20260501 08:11:25.810871 1986 ts_tablet_manager.cc:1403] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260501 08:11:25.811020 1986 raft_consensus.cc:359] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 12 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.811113 1986 raft_consensus.cc:740] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254a82ec6f2e495cbfb6d4f83d19a0f6, State: Initialized, Role: FOLLOWER
I20260501 08:11:25.811213 1986 consensus_queue.cc:260] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 23, Last appended: 12.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.811401 1986 ts_tablet_manager.cc:1434] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260501 08:11:25.812676 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.1:39211
I20260501 08:11:25.815169 2045 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.1:39211 every 8 connection(s)
W20260501 08:11:25.815623 2046 heartbeater.cc:646] Failed to heartbeat to 127.1.19.62:39201 (0 consecutive failures): Network error: Failed to ping master at 127.1.19.62:39201: Client connection negotiation failed: client connection to 127.1.19.62:39201: connect: Connection refused (error 111)
W20260501 08:11:25.816308 2046 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260501 08:11:25.822548 1100 tablet_server.cc:179] TabletServer@127.1.19.1:39211 shutting down...
I20260501 08:11:25.903035 2024 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:25.903151 2024 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 12 FOLLOWER]: Advancing to term 13
I20260501 08:11:25.903965 2024 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:25.904250 2024 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 13 election: Requested vote from peers 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
W20260501 08:11:25.905125 1980 leader_election.cc:336] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551): Network error: Client connection negotiation failed: client connection to 127.1.19.2:46551: connect: Connection refused (error 111)
W20260501 08:11:25.905225 1981 leader_election.cc:336] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871): Network error: Client connection negotiation failed: client connection to 127.1.19.3:37871: connect: Connection refused (error 111)
I20260501 08:11:25.905306 1981 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 254a82ec6f2e495cbfb6d4f83d19a0f6; no voters: 5cedf472a6394d4296c037d18340cb1e, a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:25.905404 2024 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260501 08:11:26.024201 2024 raft_consensus.cc:493] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:26.024307 2024 raft_consensus.cc:3060] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 13 FOLLOWER]: Advancing to term 14
I20260501 08:11:26.024984 2024 raft_consensus.cc:515] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254a82ec6f2e495cbfb6d4f83d19a0f6" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 39211 } } peers { permanent_uuid: "5cedf472a6394d4296c037d18340cb1e" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 37871 } } peers { permanent_uuid: "a0a0ccfc23374b689a7cebe47834418b" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 46551 } }
I20260501 08:11:26.025182 2024 leader_election.cc:290] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 14 election: Requested vote from peers 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871), a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551)
W20260501 08:11:26.025950 1981 leader_election.cc:336] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 5cedf472a6394d4296c037d18340cb1e (127.1.19.3:37871): Network error: Client connection negotiation failed: client connection to 127.1.19.3:37871: connect: Connection refused (error 111)
W20260501 08:11:26.026037 1980 leader_election.cc:336] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer a0a0ccfc23374b689a7cebe47834418b (127.1.19.2:46551): Network error: Client connection negotiation failed: client connection to 127.1.19.2:46551: connect: Connection refused (error 111)
I20260501 08:11:26.026118 1980 leader_election.cc:304] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 254a82ec6f2e495cbfb6d4f83d19a0f6; no voters: 5cedf472a6394d4296c037d18340cb1e, a0a0ccfc23374b689a7cebe47834418b
I20260501 08:11:26.026222 2024 raft_consensus.cc:2749] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260501 08:11:26.065670 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:26.066035 1100 tablet_replica.cc:333] stopping tablet replica
I20260501 08:11:26.066123 1100 raft_consensus.cc:2243] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 14 FOLLOWER]: Raft consensus shutting down.
I20260501 08:11:26.066181 1100 raft_consensus.cc:2272] T b075b51cc8d04cc5bb00758973fbb343 P 254a82ec6f2e495cbfb6d4f83d19a0f6 [term 14 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:26.078526 1100 tablet_server.cc:196] TabletServer@127.1.19.1:39211 shutdown complete.
[ OK ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap (2670 ms)
[ RUN ] TimestampAdvancementITest.Kudu2463Test
I20260501 08:11:26.083706 1100 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.19.62:33213
I20260501 08:11:26.083943 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:26.085270 2053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:26.085390 2054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:26.085399 2056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:26.085559 1100 server_base.cc:1061] running on GCE node
I20260501 08:11:26.085655 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:26.085688 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:26.085707 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623086085707 us; error 0 us; skew 500 ppm
I20260501 08:11:26.086328 1100 webserver.cc:492] Webserver started at http://127.1.19.62:39137/ using document root <none> and password file <none>
I20260501 08:11:26.086398 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:26.086431 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:26.086459 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:26.086740 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/master-0-root/instance:
uuid: "123130273061429aad1445d0a94ba0ca"
format_stamp: "Formatted at 2026-05-01 08:11:26 on dist-test-slave-pv7w"
I20260501 08:11:26.087781 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:26.088491 2061 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.088663 1100 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260501 08:11:26.088732 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/master-0-root
uuid: "123130273061429aad1445d0a94ba0ca"
format_stamp: "Formatted at 2026-05-01 08:11:26 on dist-test-slave-pv7w"
I20260501 08:11:26.088817 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:26.100082 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:26.100560 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:26.103242 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.62:33213
I20260501 08:11:26.103834 2113 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.62:33213 every 8 connection(s)
I20260501 08:11:26.103984 2114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:11:26.105018 2114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca: Bootstrap starting.
I20260501 08:11:26.105280 2114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca: Neither blocks nor log segments found. Creating new log.
I20260501 08:11:26.105744 2114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca: No bootstrap required, opened a new log
I20260501 08:11:26.105849 2114 raft_consensus.cc:359] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123130273061429aad1445d0a94ba0ca" member_type: VOTER }
I20260501 08:11:26.105896 2114 raft_consensus.cc:385] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:11:26.105911 2114 raft_consensus.cc:740] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123130273061429aad1445d0a94ba0ca, State: Initialized, Role: FOLLOWER
I20260501 08:11:26.105993 2114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123130273061429aad1445d0a94ba0ca" member_type: VOTER }
I20260501 08:11:26.106063 2114 raft_consensus.cc:399] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 08:11:26.106092 2114 raft_consensus.cc:493] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 08:11:26.106122 2114 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:11:26.106628 2114 raft_consensus.cc:515] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123130273061429aad1445d0a94ba0ca" member_type: VOTER }
I20260501 08:11:26.106709 2114 leader_election.cc:304] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 123130273061429aad1445d0a94ba0ca; no voters:
I20260501 08:11:26.106830 2114 leader_election.cc:290] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 08:11:26.106956 2117 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260501 08:11:26.107033 2114 sys_catalog.cc:565] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260501 08:11:26.107138 2117 raft_consensus.cc:697] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 1 LEADER]: Becoming Leader. State: Replica: 123130273061429aad1445d0a94ba0ca, State: Running, Role: LEADER
I20260501 08:11:26.107221 2117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123130273061429aad1445d0a94ba0ca" member_type: VOTER }
I20260501 08:11:26.107523 2118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "123130273061429aad1445d0a94ba0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123130273061429aad1445d0a94ba0ca" member_type: VOTER } }
I20260501 08:11:26.107606 2118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [sys.catalog]: This master's current role is: LEADER
I20260501 08:11:26.107554 2119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 123130273061429aad1445d0a94ba0ca. Latest consensus state: current_term: 1 leader_uuid: "123130273061429aad1445d0a94ba0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123130273061429aad1445d0a94ba0ca" member_type: VOTER } }
I20260501 08:11:26.107657 2119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [sys.catalog]: This master's current role is: LEADER
I20260501 08:11:26.108063 2125 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 08:11:26.108265 2125 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 08:11:26.108484 1100 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260501 08:11:26.109048 2125 catalog_manager.cc:1357] Generated new cluster ID: 40946904e7e1480d806b84f0d89f91df
I20260501 08:11:26.109092 2125 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 08:11:26.142511 2125 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 08:11:26.142944 2125 catalog_manager.cc:1514] Loading token signing keys...
I20260501 08:11:26.152751 2125 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca: Generated new TSK 0
I20260501 08:11:26.152846 2125 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 08:11:26.172663 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:26.174338 2135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:26.174419 2138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:26.174448 2136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:26.174561 1100 server_base.cc:1061] running on GCE node
I20260501 08:11:26.174782 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:26.174815 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:26.174829 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623086174829 us; error 0 us; skew 500 ppm
I20260501 08:11:26.175537 1100 webserver.cc:492] Webserver started at http://127.1.19.1:36665/ using document root <none> and password file <none>
I20260501 08:11:26.175621 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:26.175700 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:26.175776 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:26.176231 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-0-root/instance:
uuid: "9cba4effa8624605bc8bdf29e2c39e76"
format_stamp: "Formatted at 2026-05-01 08:11:26 on dist-test-slave-pv7w"
I20260501 08:11:26.177232 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260501 08:11:26.177735 2143 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.177948 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260501 08:11:26.177990 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-0-root
uuid: "9cba4effa8624605bc8bdf29e2c39e76"
format_stamp: "Formatted at 2026-05-01 08:11:26 on dist-test-slave-pv7w"
I20260501 08:11:26.178051 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:26.193265 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:26.193536 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:26.193710 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:26.193943 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:11:26.193970 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.193993 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:11:26.194046 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.197024 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.1:38547
I20260501 08:11:26.197044 2206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.1:38547 every 8 connection(s)
I20260501 08:11:26.197435 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:26.198922 2212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:26.199015 2214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:26.199059 1100 server_base.cc:1061] running on GCE node
W20260501 08:11:26.198796 2211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:26.199797 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:26.199854 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:26.199875 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623086199875 us; error 0 us; skew 500 ppm
I20260501 08:11:26.200613 1100 webserver.cc:492] Webserver started at http://127.1.19.2:43103/ using document root <none> and password file <none>
I20260501 08:11:26.200713 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:26.200765 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:26.200814 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:26.201151 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-1-root/instance:
uuid: "18c17f24039d4960b18af6eef566821d"
format_stamp: "Formatted at 2026-05-01 08:11:26 on dist-test-slave-pv7w"
I20260501 08:11:26.201400 2207 heartbeater.cc:344] Connected to a master server at 127.1.19.62:33213
I20260501 08:11:26.201468 2207 heartbeater.cc:461] Registering TS with master...
I20260501 08:11:26.201591 2207 heartbeater.cc:507] Master 127.1.19.62:33213 requested a full tablet report, sending...
I20260501 08:11:26.201839 2078 ts_manager.cc:194] Registered new tserver with Master: 9cba4effa8624605bc8bdf29e2c39e76 (127.1.19.1:38547)
I20260501 08:11:26.202353 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260501 08:11:26.202562 2078 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57222
I20260501 08:11:26.202872 2219 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.202986 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.203032 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-1-root
uuid: "18c17f24039d4960b18af6eef566821d"
format_stamp: "Formatted at 2026-05-01 08:11:26 on dist-test-slave-pv7w"
I20260501 08:11:26.203078 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:26.237934 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:26.238265 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:26.238467 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:26.238696 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:11:26.238725 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.238749 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:11:26.238762 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.242918 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.2:42973
I20260501 08:11:26.243000 2282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.2:42973 every 8 connection(s)
I20260501 08:11:26.243388 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:26.244777 2286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:26.244944 2289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:26.245057 1100 server_base.cc:1061] running on GCE node
W20260501 08:11:26.245198 2287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:26.245412 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:26.245455 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:26.245468 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623086245468 us; error 0 us; skew 500 ppm
I20260501 08:11:26.246186 1100 webserver.cc:492] Webserver started at http://127.1.19.3:43239/ using document root <none> and password file <none>
I20260501 08:11:26.246286 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:26.246337 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:26.246383 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:26.246678 2283 heartbeater.cc:344] Connected to a master server at 127.1.19.62:33213
I20260501 08:11:26.246742 2283 heartbeater.cc:461] Registering TS with master...
I20260501 08:11:26.246716 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-2-root/instance:
uuid: "9e7c0c588a3f44438084a39966ce413b"
format_stamp: "Formatted at 2026-05-01 08:11:26 on dist-test-slave-pv7w"
I20260501 08:11:26.246862 2283 heartbeater.cc:507] Master 127.1.19.62:33213 requested a full tablet report, sending...
I20260501 08:11:26.247092 2078 ts_manager.cc:194] Registered new tserver with Master: 18c17f24039d4960b18af6eef566821d (127.1.19.2:42973)
I20260501 08:11:26.247467 2078 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57226
I20260501 08:11:26.247762 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:26.248369 2294 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.248602 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260501 08:11:26.248672 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-2-root
uuid: "9e7c0c588a3f44438084a39966ce413b"
format_stamp: "Formatted at 2026-05-01 08:11:26 on dist-test-slave-pv7w"
I20260501 08:11:26.248749 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:26.269188 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:26.269516 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:26.269711 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:26.269935 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:11:26.269984 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.270018 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:11:26.270164 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:26.273645 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.3:35033
I20260501 08:11:26.273741 2357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.3:35033 every 8 connection(s)
I20260501 08:11:26.277568 2358 heartbeater.cc:344] Connected to a master server at 127.1.19.62:33213
I20260501 08:11:26.277637 2358 heartbeater.cc:461] Registering TS with master...
I20260501 08:11:26.277765 2358 heartbeater.cc:507] Master 127.1.19.62:33213 requested a full tablet report, sending...
I20260501 08:11:26.278119 2078 ts_manager.cc:194] Registered new tserver with Master: 9e7c0c588a3f44438084a39966ce413b (127.1.19.3:35033)
I20260501 08:11:26.278399 1100 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004510554s
I20260501 08:11:26.278586 2078 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57236
I20260501 08:11:26.292974 2078 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57282:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 08:11:26.293190 2078 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 08:11:26.298563 2241 tablet_service.cc:1511] Processing CreateTablet for tablet d4ba44fb3f014c57bea6eb65334fce5b (DEFAULT_TABLE table=test-workload [id=4ca4d88f173b492fb9510249d2dab767]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 08:11:26.298740 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ba44fb3f014c57bea6eb65334fce5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:11:26.298980 2171 tablet_service.cc:1511] Processing CreateTablet for tablet d4ba44fb3f014c57bea6eb65334fce5b (DEFAULT_TABLE table=test-workload [id=4ca4d88f173b492fb9510249d2dab767]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 08:11:26.299078 2171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ba44fb3f014c57bea6eb65334fce5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:11:26.300402 2322 tablet_service.cc:1511] Processing CreateTablet for tablet d4ba44fb3f014c57bea6eb65334fce5b (DEFAULT_TABLE table=test-workload [id=4ca4d88f173b492fb9510249d2dab767]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 08:11:26.300451 2380 tablet_bootstrap.cc:492] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: Bootstrap starting.
I20260501 08:11:26.300547 2322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ba44fb3f014c57bea6eb65334fce5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:11:26.300395 2379 tablet_bootstrap.cc:492] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d: Bootstrap starting.
I20260501 08:11:26.301173 2380 tablet_bootstrap.cc:654] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: Neither blocks nor log segments found. Creating new log.
I20260501 08:11:26.301776 2380 tablet_bootstrap.cc:492] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: No bootstrap required, opened a new log
I20260501 08:11:26.301831 2380 ts_tablet_manager.cc:1403] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:26.302001 2380 raft_consensus.cc:359] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.302088 2380 raft_consensus.cc:385] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:11:26.302114 2380 raft_consensus.cc:740] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cba4effa8624605bc8bdf29e2c39e76, State: Initialized, Role: FOLLOWER
I20260501 08:11:26.302187 2380 consensus_queue.cc:260] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.302342 2382 tablet_bootstrap.cc:492] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b: Bootstrap starting.
I20260501 08:11:26.302407 2207 heartbeater.cc:499] Master 127.1.19.62:33213 was elected leader, sending a full tablet report...
I20260501 08:11:26.302364 2380 ts_tablet_manager.cc:1434] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260501 08:11:26.302953 2379 tablet_bootstrap.cc:654] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d: Neither blocks nor log segments found. Creating new log.
I20260501 08:11:26.303020 2382 tablet_bootstrap.cc:654] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b: Neither blocks nor log segments found. Creating new log.
I20260501 08:11:26.303781 2379 tablet_bootstrap.cc:492] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d: No bootstrap required, opened a new log
I20260501 08:11:26.303838 2379 ts_tablet_manager.cc:1403] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260501 08:11:26.303889 2382 tablet_bootstrap.cc:492] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b: No bootstrap required, opened a new log
I20260501 08:11:26.303943 2382 ts_tablet_manager.cc:1403] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260501 08:11:26.304080 2379 raft_consensus.cc:359] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.304162 2382 raft_consensus.cc:359] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.304190 2379 raft_consensus.cc:385] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:11:26.304239 2382 raft_consensus.cc:385] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:11:26.304247 2379 raft_consensus.cc:740] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c17f24039d4960b18af6eef566821d, State: Initialized, Role: FOLLOWER
I20260501 08:11:26.304271 2382 raft_consensus.cc:740] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7c0c588a3f44438084a39966ce413b, State: Initialized, Role: FOLLOWER
I20260501 08:11:26.304332 2379 consensus_queue.cc:260] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.304356 2382 consensus_queue.cc:260] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.304497 2379 ts_tablet_manager.cc:1434] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:26.304528 2283 heartbeater.cc:499] Master 127.1.19.62:33213 was elected leader, sending a full tablet report...
I20260501 08:11:26.304541 2382 ts_tablet_manager.cc:1434] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:26.304621 2358 heartbeater.cc:499] Master 127.1.19.62:33213 was elected leader, sending a full tablet report...
I20260501 08:11:26.432787 2387 raft_consensus.cc:493] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 08:11:26.432940 2387 raft_consensus.cc:515] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.433246 2387 leader_election.cc:290] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cba4effa8624605bc8bdf29e2c39e76 (127.1.19.1:38547), 18c17f24039d4960b18af6eef566821d (127.1.19.2:42973)
I20260501 08:11:26.436642 2181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4ba44fb3f014c57bea6eb65334fce5b" candidate_uuid: "9e7c0c588a3f44438084a39966ce413b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cba4effa8624605bc8bdf29e2c39e76" is_pre_election: true
I20260501 08:11:26.436765 2181 raft_consensus.cc:2468] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e7c0c588a3f44438084a39966ce413b in term 0.
I20260501 08:11:26.436924 2296 leader_election.cc:304] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cba4effa8624605bc8bdf29e2c39e76, 9e7c0c588a3f44438084a39966ce413b; no voters:
I20260501 08:11:26.437043 2387 raft_consensus.cc:2804] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 08:11:26.437081 2387 raft_consensus.cc:493] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 08:11:26.437110 2387 raft_consensus.cc:3060] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:11:26.437167 2257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4ba44fb3f014c57bea6eb65334fce5b" candidate_uuid: "9e7c0c588a3f44438084a39966ce413b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c17f24039d4960b18af6eef566821d" is_pre_election: true
I20260501 08:11:26.437235 2257 raft_consensus.cc:2468] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e7c0c588a3f44438084a39966ce413b in term 0.
I20260501 08:11:26.437817 2387 raft_consensus.cc:515] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.437958 2387 leader_election.cc:290] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [CANDIDATE]: Term 1 election: Requested vote from peers 9cba4effa8624605bc8bdf29e2c39e76 (127.1.19.1:38547), 18c17f24039d4960b18af6eef566821d (127.1.19.2:42973)
I20260501 08:11:26.438294 2257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4ba44fb3f014c57bea6eb65334fce5b" candidate_uuid: "9e7c0c588a3f44438084a39966ce413b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c17f24039d4960b18af6eef566821d"
I20260501 08:11:26.438380 2181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4ba44fb3f014c57bea6eb65334fce5b" candidate_uuid: "9e7c0c588a3f44438084a39966ce413b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cba4effa8624605bc8bdf29e2c39e76"
I20260501 08:11:26.438395 2257 raft_consensus.cc:3060] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:11:26.438484 2181 raft_consensus.cc:3060] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:11:26.439232 2181 raft_consensus.cc:2468] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e7c0c588a3f44438084a39966ce413b in term 1.
I20260501 08:11:26.439241 2257 raft_consensus.cc:2468] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e7c0c588a3f44438084a39966ce413b in term 1.
I20260501 08:11:26.439414 2296 leader_election.cc:304] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cba4effa8624605bc8bdf29e2c39e76, 9e7c0c588a3f44438084a39966ce413b; no voters:
I20260501 08:11:26.439550 2387 raft_consensus.cc:2804] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 FOLLOWER]: Leader election won for term 1
I20260501 08:11:26.439702 2387 raft_consensus.cc:697] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Becoming Leader. State: Replica: 9e7c0c588a3f44438084a39966ce413b, State: Running, Role: LEADER
I20260501 08:11:26.439747 2387 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.440456 2077 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e7c0c588a3f44438084a39966ce413b (127.1.19.3). New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.448418 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 08:11:26.448436 2257 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 08:11:26.448814 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 08:11:26.449126 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 08:11:26.450577 2396 mvcc.cc:204] Tried to move back new op lower bound from 7281144162089422848 to 7281144162057764864. Current Snapshot: MvccSnapshot[applied={T|T < 7281144162089422848}]
I20260501 08:11:26.471024 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.471541 2257 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20260501 08:11:26.471638 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20260501 08:11:26.471747 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260501 08:11:26.471855 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260501 08:11:26.472602 2389 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.472711 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.473938 2078 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index -1 to 17, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.474007 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.474357 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20260501 08:11:26.474359 2257 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Committing config change with OpId 1.17: config changed from index -1 to 17, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
W20260501 08:11:26.474709 2078 catalog_manager.cc:5349] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4ba44fb3f014c57bea6eb65334fce5b with cas_config_opid_index 17: no extra replica candidate found for tablet d4ba44fb3f014c57bea6eb65334fce5b (table test-workload [id=4ca4d88f173b492fb9510249d2dab767]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20260501 08:11:26.474687 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260501 08:11:26.475196 2387 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.475319 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.475669 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20260501 08:11:26.476344 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260501 08:11:26.476402 2075 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index 17 to 18, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.476725 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.476729 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.477347 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20260501 08:11:26.477536 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260501 08:11:26.477681 2257 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20260501 08:11:26.477973 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260501 08:11:26.478094 2387 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.478343 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.479204 2078 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index 18 to 19, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.479395 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.479499 2257 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.479761 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20260501 08:11:26.479919 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260501 08:11:26.480504 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20260501 08:11:26.480711 2389 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.480793 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260501 08:11:26.480824 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.481060 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.481812 2075 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index 19 to 20, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.481854 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.482268 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20260501 08:11:26.482460 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20260501 08:11:26.482537 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260501 08:11:26.482643 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260501 08:11:26.483244 2413 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.483381 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.483417 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.484457 2077 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 reported cstate change: config changed from index 20 to 21, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.484563 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.484961 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20260501 08:11:26.484994 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20260501 08:11:26.485150 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260501 08:11:26.485287 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260501 08:11:26.485911 2413 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.485975 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.486068 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.486948 2078 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d reported cstate change: config changed from index 21 to 22, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.487111 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.487608 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20260501 08:11:26.487615 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20260501 08:11:26.487905 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260501 08:11:26.487998 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260501 08:11:26.488552 2387 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.488667 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.488684 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.489534 2077 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 reported cstate change: config changed from index 22 to 23, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.490226 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.490594 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20260501 08:11:26.490734 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20260501 08:11:26.490850 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260501 08:11:26.490954 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260501 08:11:26.491472 2413 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.491631 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.491690 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.492607 2075 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 reported cstate change: config changed from index 23 to 24, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.492743 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.493119 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20260501 08:11:26.493296 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260501 08:11:26.493566 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20260501 08:11:26.493846 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260501 08:11:26.493997 2387 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.494462 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.495039 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.495128 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.495151 2078 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index 24 to 25, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.495870 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20260501 08:11:26.495960 2180 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20260501 08:11:26.496088 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260501 08:11:26.496243 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260501 08:11:26.496694 2387 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.496804 2180 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.496912 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.497798 2078 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index 25 to 26, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.498663 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.499145 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20260501 08:11:26.499212 2180 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20260501 08:11:26.499312 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260501 08:11:26.499454 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260501 08:11:26.500051 2413 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.500200 2180 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.501224 2075 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 reported cstate change: config changed from index 26 to 27, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.501390 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.501422 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.501844 2180 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20260501 08:11:26.502193 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260501 08:11:26.502753 2389 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.502880 2257 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20260501 08:11:26.503188 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260501 08:11:26.503484 2257 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.503860 2180 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.503944 2077 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index 27 to 28, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.504006 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.505175 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20260501 08:11:26.505301 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20260501 08:11:26.505460 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260501 08:11:26.505527 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260501 08:11:26.506076 2389 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.506179 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.507089 2077 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 reported cstate change: config changed from index 28 to 29, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.507145 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.507481 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20260501 08:11:26.507545 2257 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20260501 08:11:26.507685 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20260501 08:11:26.507813 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20260501 08:11:26.507778 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.508208 2387 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.508966 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.509093 2064 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4ba44fb3f014c57bea6eb65334fce5b with cas_config_opid_index 19: aborting the task: latest config opid_index 29; task opid_index 19
I20260501 08:11:26.509500 2075 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index 29 to 30, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.509603 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.509583 2257 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.510057 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20260501 08:11:26.510332 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20260501 08:11:26.510946 2387 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.511041 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.511174 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20260501 08:11:26.511708 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20260501 08:11:26.512012 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.512049 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.512230 2075 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 reported cstate change: config changed from index 30 to 31, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.513038 2257 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20260501 08:11:26.513150 2180 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20260501 08:11:26.513304 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20260501 08:11:26.513589 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20260501 08:11:26.513834 2389 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.514029 2257 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.514273 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.514884 2075 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d reported cstate change: config changed from index 31 to 32, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.514963 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.515489 2257 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20260501 08:11:26.515767 2180 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20260501 08:11:26.515767 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20260501 08:11:26.515967 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20260501 08:11:26.516625 2413 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.516733 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.517642 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.517681 2257 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.517932 2077 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index 32 to 33, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.518464 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20260501 08:11:26.518698 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20260501 08:11:26.519121 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20260501 08:11:26.519241 2413 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.519353 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.519814 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20260501 08:11:26.520305 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.520421 2077 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 reported cstate change: config changed from index 33 to 34, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.520848 2257 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20260501 08:11:26.521075 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.521085 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.000s
I20260501 08:11:26.522033 2257 raft_consensus.cc:1217] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.33->[1.34-1.35] Dedup: 1.34->[1.35-1.35]
I20260501 08:11:26.522068 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20260501 08:11:26.522382 2387 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20260501 08:11:26.523234 2389 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.523360 2181 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.524286 2075 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index 34 to 35, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.524307 2332 consensus_queue.cc:237] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:26.524405 2257 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from VOTER to NON_VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: NON_VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.524690 2181 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20260501 08:11:26.524895 2389 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20260501 08:11:26.525372 2256 raft_consensus.cc:1275] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 LEARNER]: Refusing update from remote peer 9e7c0c588a3f44438084a39966ce413b: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20260501 08:11:26.525563 2387 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.525563 2413 consensus_queue.cc:1048] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20260501 08:11:26.525624 2180 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.525949 2256 raft_consensus.cc:2955] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } } }
I20260501 08:11:26.526522 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:26.526599 2075 catalog_manager.cc:5671] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b reported cstate change: config changed from index 35 to 36, 18c17f24039d4960b18af6eef566821d (127.1.19.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e7c0c588a3f44438084a39966ce413b" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:26.526844 2062 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4ba44fb3f014c57bea6eb65334fce5b with cas_config_opid_index 17: aborting the task: latest config opid_index 35; task opid_index 17
I20260501 08:11:26.527680 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:26.529901 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:26.534065 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:26.534086 2065 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4ba44fb3f014c57bea6eb65334fce5b with cas_config_opid_index 23: aborting the task: latest config opid_index 36; task opid_index 23
I20260501 08:11:26.536696 2063 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4ba44fb3f014c57bea6eb65334fce5b with cas_config_opid_index 25: aborting the task: latest config opid_index 36; task opid_index 25
I20260501 08:11:26.539511 2064 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4ba44fb3f014c57bea6eb65334fce5b with cas_config_opid_index 29: aborting the task: latest config opid_index 36; task opid_index 29
I20260501 08:11:26.542317 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:26.544975 2063 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4ba44fb3f014c57bea6eb65334fce5b with cas_config_opid_index 21: aborting the task: latest config opid_index 36; task opid_index 21
I20260501 08:11:26.547498 2065 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4ba44fb3f014c57bea6eb65334fce5b with cas_config_opid_index 35: aborting the task: latest config opid_index 36; task opid_index 35
I20260501 08:11:26.558576 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:26.560269 2063 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4ba44fb3f014c57bea6eb65334fce5b with cas_config_opid_index 27: aborting the task: latest config opid_index 36; task opid_index 27
I20260501 08:11:26.573125 2062 catalog_manager.cc:5218] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4ba44fb3f014c57bea6eb65334fce5b with cas_config_opid_index 33: aborting the task: latest config opid_index 36; task opid_index 33
I20260501 08:11:26.590865 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:26.655150 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:26.783440 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:27.039729 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:27.552074 1100 timestamp_advancement-itest.cc:267] GCing logs...
I20260501 08:11:27.552224 1100 log_reader.cc:385] T d4ba44fb3f014c57bea6eb65334fce5b: removed 1 log segments from log reader
I20260501 08:11:27.552243 1100 log.cc:1079] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: Deleting log segment in path: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-0-root/wals/d4ba44fb3f014c57bea6eb65334fce5b/wal-000000001 (ops 1-16)
I20260501 08:11:27.553125 1100 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20260501 08:11:27.553224 1100 master.cc:562] Master@127.1.19.62:33213 shutting down...
I20260501 08:11:27.602311 1100 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 1 LEADER]: Raft consensus shutting down.
I20260501 08:11:27.602433 1100 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:27.602455 1100 tablet_replica.cc:333] T 00000000000000000000000000000000 P 123130273061429aad1445d0a94ba0ca: stopping tablet replica
I20260501 08:11:27.615252 1100 master.cc:584] Master@127.1.19.62:33213 shutdown complete.
I20260501 08:11:27.616835 1100 tablet_server.cc:179] TabletServer@127.1.19.1:0 shutting down...
I20260501 08:11:27.698686 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:27.699072 1100 tablet_replica.cc:333] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: stopping tablet replica
I20260501 08:11:27.699141 1100 raft_consensus.cc:2243] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 08:11:27.699266 1100 raft_consensus.cc:2272] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:27.711385 1100 tablet_server.cc:196] TabletServer@127.1.19.1:0 shutdown complete.
I20260501 08:11:27.712930 1100 tablet_server.cc:179] TabletServer@127.1.19.2:0 shutting down...
W20260501 08:11:27.737289 2296 consensus_peers.cc:597] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b -> Peer 9cba4effa8624605bc8bdf29e2c39e76 (127.1.19.1:38547): Couldn't send request to peer 9cba4effa8624605bc8bdf29e2c39e76. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.1:38547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260501 08:11:27.744329 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:27.744467 1100 tablet_replica.cc:333] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d: stopping tablet replica
I20260501 08:11:27.744530 1100 raft_consensus.cc:2243] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 08:11:27.744602 1100 raft_consensus.cc:2272] T d4ba44fb3f014c57bea6eb65334fce5b P 18c17f24039d4960b18af6eef566821d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:27.757025 1100 tablet_server.cc:196] TabletServer@127.1.19.2:0 shutdown complete.
I20260501 08:11:27.758075 1100 tablet_server.cc:179] TabletServer@127.1.19.3:0 shutting down...
I20260501 08:11:27.775142 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:27.775336 1100 tablet_replica.cc:333] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b: stopping tablet replica
I20260501 08:11:27.775400 1100 raft_consensus.cc:2243] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 LEADER]: Raft consensus shutting down.
I20260501 08:11:27.775468 1100 raft_consensus.cc:2272] T d4ba44fb3f014c57bea6eb65334fce5b P 9e7c0c588a3f44438084a39966ce413b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:27.787612 1100 tablet_server.cc:196] TabletServer@127.1.19.3:0 shutdown complete.
I20260501 08:11:27.788674 1100 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20260501 08:11:27.788789 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:27.790313 2417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:27.790341 1100 server_base.cc:1061] running on GCE node
W20260501 08:11:27.790416 2419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:27.790426 2416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:27.790621 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:27.790664 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:27.790681 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623087790682 us; error 0 us; skew 500 ppm
I20260501 08:11:27.791363 1100 webserver.cc:492] Webserver started at http://127.1.19.1:36665/ using document root <none> and password file <none>
I20260501 08:11:27.791467 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:27.791525 1100 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260501 08:11:27.792357 1100 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260501 08:11:27.794166 2427 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:27.794417 1100 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260501 08:11:27.794540 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-0-root
uuid: "9cba4effa8624605bc8bdf29e2c39e76"
format_stamp: "Formatted at 2026-05-01 08:11:26 on dist-test-slave-pv7w"
I20260501 08:11:27.794642 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1777623050595939-1100-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5294
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:27.807363 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:27.807646 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:27.807835 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:27.808257 2434 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260501 08:11:27.808876 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260501 08:11:27.808950 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260501 08:11:27.809000 1100 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260501 08:11:27.809399 1100 ts_tablet_manager.cc:616] Registered 1 tablets
I20260501 08:11:27.809453 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:27.809466 2434 tablet_bootstrap.cc:492] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: Bootstrap starting.
I20260501 08:11:27.811478 2434 tablet_bootstrap.cc:492] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 08:11:27.811755 2434 tablet_bootstrap.cc:492] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: Bootstrap complete.
I20260501 08:11:27.811877 2434 ts_tablet_manager.cc:1403] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260501 08:11:27.812024 2434 raft_consensus.cc:359] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:27.812114 2434 raft_consensus.cc:740] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cba4effa8624605bc8bdf29e2c39e76, State: Initialized, Role: FOLLOWER
I20260501 08:11:27.812209 2434 consensus_queue.cc:260] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "9cba4effa8624605bc8bdf29e2c39e76" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 38547 } } peers { permanent_uuid: "18c17f24039d4960b18af6eef566821d" member_type: VOTER last_known_addr { host: "127.1.19.2" port: 42973 } } peers { permanent_uuid: "9e7c0c588a3f44438084a39966ce413b" member_type: VOTER last_known_addr { host: "127.1.19.3" port: 35033 } }
I20260501 08:11:27.812381 2434 ts_tablet_manager.cc:1434] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20260501 08:11:27.813319 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.1:38547
I20260501 08:11:27.815954 2493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.1:38547 every 8 connection(s)
W20260501 08:11:27.816885 2494 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260501 08:11:27.825435 2447 tablet_service.cc:3018] Rejecting scan request for tablet d4ba44fb3f014c57bea6eb65334fce5b: Uninitialized: clean time has not yet been initialized
I20260501 08:11:27.825802 1100 timestamp_advancement-itest.cc:447] Got response: error { code: TABLET_NOT_RUNNING status { code: UNINITIALIZED message: "clean time has not yet been initialized" } }
I20260501 08:11:27.825930 1100 tablet_server.cc:179] TabletServer@127.1.19.1:38547 shutting down...
I20260501 08:11:28.066459 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:28.066700 1100 tablet_replica.cc:333] stopping tablet replica
I20260501 08:11:28.066761 1100 raft_consensus.cc:2243] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 08:11:28.066802 1100 raft_consensus.cc:2272] T d4ba44fb3f014c57bea6eb65334fce5b P 9cba4effa8624605bc8bdf29e2c39e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:28.079198 1100 tablet_server.cc:196] TabletServer@127.1.19.1:38547 shutdown complete.
[ OK ] TimestampAdvancementITest.Kudu2463Test (2000 ms)
[ RUN ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp
I20260501 08:11:28.083974 1100 timestamp_advancement-itest.cc:460] Running with 1 replicas
I20260501 08:11:28.084219 1100 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.19.62:46779
I20260501 08:11:28.084398 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:28.085762 2499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:28.085760 2502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:28.085909 2500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:28.085999 1100 server_base.cc:1061] running on GCE node
I20260501 08:11:28.086102 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:28.086127 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:28.086139 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623088086140 us; error 0 us; skew 500 ppm
I20260501 08:11:28.086642 1100 webserver.cc:492] Webserver started at http://127.1.19.62:42915/ using document root <none> and password file <none>
I20260501 08:11:28.086761 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:28.086824 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:28.086895 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:28.087222 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/master-0-root/instance:
uuid: "fc4a3df99bb446bf92697b4da34c587c"
format_stamp: "Formatted at 2026-05-01 08:11:28 on dist-test-slave-pv7w"
I20260501 08:11:28.088238 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260501 08:11:28.088733 2507 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.088918 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.088971 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/master-0-root
uuid: "fc4a3df99bb446bf92697b4da34c587c"
format_stamp: "Formatted at 2026-05-01 08:11:28 on dist-test-slave-pv7w"
I20260501 08:11:28.089011 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:28.094388 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:28.094578 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:28.097640 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.62:46779
I20260501 08:11:28.098457 2559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.62:46779 every 8 connection(s)
I20260501 08:11:28.098611 2560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:11:28.099669 2560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c: Bootstrap starting.
I20260501 08:11:28.099953 2560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c: Neither blocks nor log segments found. Creating new log.
I20260501 08:11:28.100508 2560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c: No bootstrap required, opened a new log
I20260501 08:11:28.100620 2560 raft_consensus.cc:359] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER }
I20260501 08:11:28.100669 2560 raft_consensus.cc:385] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:11:28.100683 2560 raft_consensus.cc:740] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4a3df99bb446bf92697b4da34c587c, State: Initialized, Role: FOLLOWER
I20260501 08:11:28.100719 2560 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER }
I20260501 08:11:28.100741 2560 raft_consensus.cc:399] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 08:11:28.100754 2560 raft_consensus.cc:493] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 08:11:28.100811 2560 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:11:28.101337 2560 raft_consensus.cc:515] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER }
I20260501 08:11:28.101394 2560 leader_election.cc:304] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc4a3df99bb446bf92697b4da34c587c; no voters:
I20260501 08:11:28.101524 2560 leader_election.cc:290] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 08:11:28.101653 2563 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 1 FOLLOWER]: Leader election won for term 1
I20260501 08:11:28.101697 2560 sys_catalog.cc:565] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [sys.catalog]: configured and running, proceeding with master startup.
I20260501 08:11:28.101843 2563 raft_consensus.cc:697] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 1 LEADER]: Becoming Leader. State: Replica: fc4a3df99bb446bf92697b4da34c587c, State: Running, Role: LEADER
I20260501 08:11:28.101905 2563 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER }
I20260501 08:11:28.102154 2564 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc4a3df99bb446bf92697b4da34c587c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER } }
I20260501 08:11:28.102186 2565 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc4a3df99bb446bf92697b4da34c587c. Latest consensus state: current_term: 1 leader_uuid: "fc4a3df99bb446bf92697b4da34c587c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER } }
I20260501 08:11:28.102373 2565 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [sys.catalog]: This master's current role is: LEADER
I20260501 08:11:28.102618 2571 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 08:11:28.102316 2564 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [sys.catalog]: This master's current role is: LEADER
I20260501 08:11:28.102771 2571 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 08:11:28.103193 1100 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260501 08:11:28.103480 2571 catalog_manager.cc:1357] Generated new cluster ID: be788d0e1cda4646b0ca47e696710080
I20260501 08:11:28.103529 2571 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 08:11:28.115687 2571 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 08:11:28.116290 2571 catalog_manager.cc:1514] Loading token signing keys...
I20260501 08:11:28.123415 2571 catalog_manager.cc:6044] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c: Generated new TSK 0
I20260501 08:11:28.123545 2571 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 08:11:28.135357 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:28.137598 2584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:28.137707 1100 server_base.cc:1061] running on GCE node
W20260501 08:11:28.137615 2582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:28.137650 2581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:28.138048 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:28.138084 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:28.138098 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623088138098 us; error 0 us; skew 500 ppm
I20260501 08:11:28.138723 1100 webserver.cc:492] Webserver started at http://127.1.19.1:35985/ using document root <none> and password file <none>
I20260501 08:11:28.138796 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:28.138875 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:28.138990 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:28.139323 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-0-root/instance:
uuid: "e0ac01e7c8574e9b999f27da25bb6a59"
format_stamp: "Formatted at 2026-05-01 08:11:28 on dist-test-slave-pv7w"
I20260501 08:11:28.140357 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:28.140944 2589 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.141124 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.141181 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-0-root
uuid: "e0ac01e7c8574e9b999f27da25bb6a59"
format_stamp: "Formatted at 2026-05-01 08:11:28 on dist-test-slave-pv7w"
I20260501 08:11:28.141252 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:28.151549 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:28.151839 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:28.152034 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:28.152294 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:11:28.152344 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.152387 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:11:28.152415 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.155336 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.1:42223
I20260501 08:11:28.155359 2652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.1:42223 every 8 connection(s)
I20260501 08:11:28.159193 2653 heartbeater.cc:344] Connected to a master server at 127.1.19.62:46779
I20260501 08:11:28.159286 2653 heartbeater.cc:461] Registering TS with master...
I20260501 08:11:28.159375 2653 heartbeater.cc:507] Master 127.1.19.62:46779 requested a full tablet report, sending...
I20260501 08:11:28.159657 2524 ts_manager.cc:194] Registered new tserver with Master: e0ac01e7c8574e9b999f27da25bb6a59 (127.1.19.1:42223)
I20260501 08:11:28.160034 1100 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004481648s
I20260501 08:11:28.160308 2524 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47870
I20260501 08:11:28.172856 2524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47886:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 08:11:28.177416 2617 tablet_service.cc:1511] Processing CreateTablet for tablet 1466353f61f14ca3a5c047e2da25646c (DEFAULT_TABLE table=test-workload [id=7221ce0ad03240fcb5a1b01c1c2e85d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 08:11:28.177596 2617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1466353f61f14ca3a5c047e2da25646c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 08:11:28.179036 2671 tablet_bootstrap.cc:492] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: Bootstrap starting.
I20260501 08:11:28.179340 2671 tablet_bootstrap.cc:654] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: Neither blocks nor log segments found. Creating new log.
I20260501 08:11:28.179816 2671 tablet_bootstrap.cc:492] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: No bootstrap required, opened a new log
I20260501 08:11:28.179862 2671 ts_tablet_manager.cc:1403] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:28.180016 2671 raft_consensus.cc:359] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 42223 } }
I20260501 08:11:28.180066 2671 raft_consensus.cc:385] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 08:11:28.180080 2671 raft_consensus.cc:740] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ac01e7c8574e9b999f27da25bb6a59, State: Initialized, Role: FOLLOWER
I20260501 08:11:28.180171 2671 consensus_queue.cc:260] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 42223 } }
I20260501 08:11:28.180223 2671 raft_consensus.cc:399] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 08:11:28.180239 2671 raft_consensus.cc:493] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 08:11:28.180253 2671 raft_consensus.cc:3060] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 0 FOLLOWER]: Advancing to term 1
I20260501 08:11:28.180869 2671 raft_consensus.cc:515] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 42223 } }
I20260501 08:11:28.180938 2671 leader_election.cc:304] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ac01e7c8574e9b999f27da25bb6a59; no voters:
I20260501 08:11:28.181056 2671 leader_election.cc:290] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 08:11:28.181100 2673 raft_consensus.cc:2804] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 08:11:28.181200 2671 ts_tablet_manager.cc:1434] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 08:11:28.181212 2673 raft_consensus.cc:697] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 1 LEADER]: Becoming Leader. State: Replica: e0ac01e7c8574e9b999f27da25bb6a59, State: Running, Role: LEADER
I20260501 08:11:28.181275 2653 heartbeater.cc:499] Master 127.1.19.62:46779 was elected leader, sending a full tablet report...
I20260501 08:11:28.181365 2673 consensus_queue.cc:237] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 42223 } }
I20260501 08:11:28.181919 2524 catalog_manager.cc:5671] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0ac01e7c8574e9b999f27da25bb6a59 (127.1.19.1). New cstate: current_term: 1 leader_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 42223 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:28.183620 1100 tablet_server.cc:179] TabletServer@127.1.19.1:0 shutting down...
I20260501 08:11:28.186043 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:28.186249 1100 tablet_replica.cc:333] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: stopping tablet replica
I20260501 08:11:28.186303 1100 raft_consensus.cc:2243] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 1 LEADER]: Raft consensus shutting down.
I20260501 08:11:28.186342 1100 raft_consensus.cc:2272] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:28.188274 1100 tablet_server.cc:196] TabletServer@127.1.19.1:0 shutdown complete.
I20260501 08:11:28.189177 1100 master.cc:562] Master@127.1.19.62:46779 shutting down...
I20260501 08:11:28.192292 1100 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 1 LEADER]: Raft consensus shutting down.
I20260501 08:11:28.192359 1100 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:28.192451 1100 tablet_replica.cc:333] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c: stopping tablet replica
I20260501 08:11:28.204854 1100 master.cc:584] Master@127.1.19.62:46779 shutdown complete.
I20260501 08:11:28.206504 1100 timestamp_advancement-itest.cc:460] Running with 3 replicas
I20260501 08:11:28.206743 1100 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.19.62:40279
I20260501 08:11:28.207001 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:28.208397 2681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:28.208426 2679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:28.208526 1100 server_base.cc:1061] running on GCE node
W20260501 08:11:28.208439 2678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:28.208760 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:28.208796 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:28.208809 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623088208809 us; error 0 us; skew 500 ppm
I20260501 08:11:28.209326 1100 webserver.cc:492] Webserver started at http://127.1.19.62:46231/ using document root <none> and password file <none>
I20260501 08:11:28.209386 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:28.209456 1100 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260501 08:11:28.210045 1100 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.210484 2686 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.210623 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.210676 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/master-0-root
uuid: "fc4a3df99bb446bf92697b4da34c587c"
format_stamp: "Formatted at 2026-05-01 08:11:28 on dist-test-slave-pv7w"
I20260501 08:11:28.210722 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:28.221311 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:28.221504 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:28.223953 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.62:40279
I20260501 08:11:28.224962 2738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.62:40279 every 8 connection(s)
I20260501 08:11:28.225543 2739 sys_catalog.cc:263] Verifying existing consensus state
I20260501 08:11:28.225869 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c: Bootstrap starting.
I20260501 08:11:28.227383 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 08:11:28.227581 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c: Bootstrap complete.
I20260501 08:11:28.227757 2739 raft_consensus.cc:359] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER }
I20260501 08:11:28.227813 2739 raft_consensus.cc:740] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4a3df99bb446bf92697b4da34c587c, State: Initialized, Role: FOLLOWER
I20260501 08:11:28.227850 2739 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER }
I20260501 08:11:28.227870 2739 raft_consensus.cc:399] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 08:11:28.227885 2739 raft_consensus.cc:493] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 08:11:28.227897 2739 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 1 FOLLOWER]: Advancing to term 2
I20260501 08:11:28.228582 2739 raft_consensus.cc:515] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER }
I20260501 08:11:28.228639 2739 leader_election.cc:304] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc4a3df99bb446bf92697b4da34c587c; no voters:
I20260501 08:11:28.228760 2739 leader_election.cc:290] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260501 08:11:28.228924 2739 sys_catalog.cc:565] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [sys.catalog]: configured and running, proceeding with master startup.
I20260501 08:11:28.228894 2742 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 2 FOLLOWER]: Leader election won for term 2
I20260501 08:11:28.229080 2742 raft_consensus.cc:697] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 2 LEADER]: Becoming Leader. State: Replica: fc4a3df99bb446bf92697b4da34c587c, State: Running, Role: LEADER
I20260501 08:11:28.229152 2742 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER }
I20260501 08:11:28.229388 2743 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fc4a3df99bb446bf92697b4da34c587c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER } }
I20260501 08:11:28.229493 2743 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [sys.catalog]: This master's current role is: LEADER
I20260501 08:11:28.229398 2744 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc4a3df99bb446bf92697b4da34c587c. Latest consensus state: current_term: 2 leader_uuid: "fc4a3df99bb446bf92697b4da34c587c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4a3df99bb446bf92697b4da34c587c" member_type: VOTER } }
I20260501 08:11:28.229650 2744 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [sys.catalog]: This master's current role is: LEADER
I20260501 08:11:28.229678 2749 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 08:11:28.229987 2749 catalog_manager.cc:679] Loaded metadata for table test-workload [id=7221ce0ad03240fcb5a1b01c1c2e85d6]
I20260501 08:11:28.230131 2749 tablet_loader.cc:96] loaded metadata for tablet 1466353f61f14ca3a5c047e2da25646c (table test-workload [id=7221ce0ad03240fcb5a1b01c1c2e85d6])
I20260501 08:11:28.230196 2749 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 08:11:28.230288 2749 catalog_manager.cc:1269] Loaded cluster ID: be788d0e1cda4646b0ca47e696710080
I20260501 08:11:28.230331 2749 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 08:11:28.230522 1100 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260501 08:11:28.231154 2749 catalog_manager.cc:1514] Loading token signing keys...
I20260501 08:11:28.231257 2749 catalog_manager.cc:6055] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c: Loaded TSK: 0
I20260501 08:11:28.231338 2749 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 08:11:28.231808 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:28.233431 2759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:28.233431 2760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:28.233539 1100 server_base.cc:1061] running on GCE node
W20260501 08:11:28.233453 2762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:28.233894 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:28.233986 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:28.234001 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623088234002 us; error 0 us; skew 500 ppm
I20260501 08:11:28.234624 1100 webserver.cc:492] Webserver started at http://127.1.19.1:36127/ using document root <none> and password file <none>
I20260501 08:11:28.234745 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:28.234779 1100 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260501 08:11:28.235431 1100 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260501 08:11:28.235985 2767 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.236171 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260501 08:11:28.236225 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-0-root
uuid: "e0ac01e7c8574e9b999f27da25bb6a59"
format_stamp: "Formatted at 2026-05-01 08:11:28 on dist-test-slave-pv7w"
I20260501 08:11:28.236280 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:28.255407 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:28.255632 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:28.255766 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:28.256690 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260501 08:11:28.256731 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260501 08:11:28.257177 1100 ts_tablet_manager.cc:616] Registered 1 tablets
I20260501 08:11:28.257207 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.257259 2774 tablet_bootstrap.cc:492] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: Bootstrap starting.
I20260501 08:11:28.258726 2774 tablet_bootstrap.cc:492] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 08:11:28.259011 2774 tablet_bootstrap.cc:492] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: Bootstrap complete.
I20260501 08:11:28.259137 2774 ts_tablet_manager.cc:1403] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260501 08:11:28.259298 2774 raft_consensus.cc:359] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 42223 } }
I20260501 08:11:28.259382 2774 raft_consensus.cc:740] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ac01e7c8574e9b999f27da25bb6a59, State: Initialized, Role: FOLLOWER
I20260501 08:11:28.259445 2774 consensus_queue.cc:260] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 42223 } }
I20260501 08:11:28.259498 2774 raft_consensus.cc:399] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 08:11:28.259534 2774 raft_consensus.cc:493] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 08:11:28.259562 2774 raft_consensus.cc:3060] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 1 FOLLOWER]: Advancing to term 2
I20260501 08:11:28.260524 2774 raft_consensus.cc:515] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 42223 } }
I20260501 08:11:28.260607 2774 leader_election.cc:304] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ac01e7c8574e9b999f27da25bb6a59; no voters:
I20260501 08:11:28.260712 2774 leader_election.cc:290] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260501 08:11:28.260769 2829 raft_consensus.cc:2804] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 2 FOLLOWER]: Leader election won for term 2
I20260501 08:11:28.260857 2774 ts_tablet_manager.cc:1434] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260501 08:11:28.260905 2829 raft_consensus.cc:697] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 2 LEADER]: Becoming Leader. State: Replica: e0ac01e7c8574e9b999f27da25bb6a59, State: Running, Role: LEADER
I20260501 08:11:28.260957 2829 consensus_queue.cc:237] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 42223 } }
I20260501 08:11:28.261053 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.1:41863
I20260501 08:11:28.261158 2834 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.1:41863 every 8 connection(s)
I20260501 08:11:28.261451 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:28.262848 2841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:28.263118 1100 server_base.cc:1061] running on GCE node
W20260501 08:11:28.262848 2842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:28.262980 2844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:28.263345 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:28.263391 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:28.263417 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623088263416 us; error 0 us; skew 500 ppm
I20260501 08:11:28.264240 1100 webserver.cc:492] Webserver started at http://127.1.19.2:40487/ using document root <none> and password file <none>
I20260501 08:11:28.264344 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:28.264412 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:28.264461 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:28.264729 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-1-root/instance:
uuid: "1277a96775df4ae0b6d2748290124f7a"
format_stamp: "Formatted at 2026-05-01 08:11:28 on dist-test-slave-pv7w"
I20260501 08:11:28.265166 2836 heartbeater.cc:344] Connected to a master server at 127.1.19.62:40279
I20260501 08:11:28.265244 2836 heartbeater.cc:461] Registering TS with master...
I20260501 08:11:28.265363 2836 heartbeater.cc:507] Master 127.1.19.62:40279 requested a full tablet report, sending...
I20260501 08:11:28.265658 2703 ts_manager.cc:194] Registered new tserver with Master: e0ac01e7c8574e9b999f27da25bb6a59 (127.1.19.1:41863)
I20260501 08:11:28.265741 2703 catalog_manager.cc:5671] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ac01e7c8574e9b999f27da25bb6a59" member_type: VOTER last_known_addr { host: "127.1.19.1" port: 42223 } health_report { overall_health: HEALTHY } } }
I20260501 08:11:28.265846 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260501 08:11:28.266348 2849 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.266527 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260501 08:11:28.266585 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-1-root
uuid: "1277a96775df4ae0b6d2748290124f7a"
format_stamp: "Formatted at 2026-05-01 08:11:28 on dist-test-slave-pv7w"
I20260501 08:11:28.266631 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:28.267144 2703 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54752
I20260501 08:11:28.267644 2836 heartbeater.cc:499] Master 127.1.19.62:40279 was elected leader, sending a full tablet report...
I20260501 08:11:28.286118 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:28.286377 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:28.286528 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:28.286751 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:11:28.286778 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.286827 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:11:28.286855 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.290267 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.2:46537
I20260501 08:11:28.290310 2913 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.2:46537 every 8 connection(s)
I20260501 08:11:28.290656 1100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 08:11:28.292255 2918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:28.292336 2920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 08:11:28.292460 2917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 08:11:28.293542 1100 server_base.cc:1061] running on GCE node
I20260501 08:11:28.293694 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260501 08:11:28.293747 1100 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260501 08:11:28.293782 1100 hybrid_clock.cc:648] HybridClock initialized: now 1777623088293781 us; error 0 us; skew 500 ppm
I20260501 08:11:28.294528 1100 webserver.cc:492] Webserver started at http://127.1.19.3:46603/ using document root <none> and password file <none>
I20260501 08:11:28.294629 1100 fs_manager.cc:362] Metadata directory not provided
I20260501 08:11:28.294620 2914 heartbeater.cc:344] Connected to a master server at 127.1.19.62:40279
I20260501 08:11:28.294708 1100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 08:11:28.294718 2914 heartbeater.cc:461] Registering TS with master...
I20260501 08:11:28.294754 1100 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 08:11:28.294842 2914 heartbeater.cc:507] Master 127.1.19.62:40279 requested a full tablet report, sending...
I20260501 08:11:28.295086 2703 ts_manager.cc:194] Registered new tserver with Master: 1277a96775df4ae0b6d2748290124f7a (127.1.19.2:46537)
I20260501 08:11:28.295192 1100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-2-root/instance:
uuid: "bafae4ac01904880a283c54cc0955698"
format_stamp: "Formatted at 2026-05-01 08:11:28 on dist-test-slave-pv7w"
I20260501 08:11:28.295492 2703 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54764
I20260501 08:11:28.296264 1100 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260501 08:11:28.296739 2925 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.296877 1100 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.296940 1100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-2-root
uuid: "bafae4ac01904880a283c54cc0955698"
format_stamp: "Formatted at 2026-05-01 08:11:28 on dist-test-slave-pv7w"
I20260501 08:11:28.297010 1100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskEWUwLy/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1777623050595939-1100-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 08:11:28.300729 1100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 08:11:28.301009 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 08:11:28.301359 1100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 08:11:28.301689 1100 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 08:11:28.301743 1100 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.301791 1100 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 08:11:28.301819 1100 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 08:11:28.305346 1100 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.3:39499
I20260501 08:11:28.305411 2988 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.3:39499 every 8 connection(s)
I20260501 08:11:28.308887 2989 heartbeater.cc:344] Connected to a master server at 127.1.19.62:40279
I20260501 08:11:28.308951 2989 heartbeater.cc:461] Registering TS with master...
I20260501 08:11:28.309093 2989 heartbeater.cc:507] Master 127.1.19.62:40279 requested a full tablet report, sending...
I20260501 08:11:28.309362 2703 ts_manager.cc:194] Registered new tserver with Master: bafae4ac01904880a283c54cc0955698 (127.1.19.3:39499)
I20260501 08:11:28.309703 2703 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54772
I20260501 08:11:28.310029 1100 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004468579s
I20260501 08:11:28.323439 1100 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20260501 08:11:28.323529 1100 tablet_server.cc:179] TabletServer@127.1.19.1:0 shutting down...
I20260501 08:11:28.326449 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:28.326637 1100 tablet_replica.cc:333] stopping tablet replica
I20260501 08:11:28.326685 1100 raft_consensus.cc:2243] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 2 LEADER]: Raft consensus shutting down.
I20260501 08:11:28.326728 1100 raft_consensus.cc:2272] T 1466353f61f14ca3a5c047e2da25646c P e0ac01e7c8574e9b999f27da25bb6a59 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:28.338865 1100 tablet_server.cc:196] TabletServer@127.1.19.1:0 shutdown complete.
I20260501 08:11:28.339951 1100 tablet_server.cc:179] TabletServer@127.1.19.2:0 shutting down...
I20260501 08:11:28.343046 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:28.354755 1100 tablet_server.cc:196] TabletServer@127.1.19.2:0 shutdown complete.
I20260501 08:11:28.355832 1100 tablet_server.cc:179] TabletServer@127.1.19.3:0 shutting down...
I20260501 08:11:28.358722 1100 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260501 08:11:28.370522 1100 tablet_server.cc:196] TabletServer@127.1.19.3:0 shutdown complete.
I20260501 08:11:28.371619 1100 master.cc:562] Master@127.1.19.62:40279 shutting down...
I20260501 08:11:28.374603 1100 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 2 LEADER]: Raft consensus shutting down.
I20260501 08:11:28.374713 1100 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fc4a3df99bb446bf92697b4da34c587c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260501 08:11:28.374812 1100 tablet_replica.cc:333] stopping tablet replica
I20260501 08:11:28.387493 1100 master.cc:584] Master@127.1.19.62:40279 shutdown complete.
[ OK ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp (308 ms)
[----------] 4 tests from TimestampAdvancementITest (37791 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (37791 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
1 FAILED TEST
I20260501 08:11:28.392454 1100 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5349: suppressed but not reported on 8 messages since previous log ~1 seconds ago
I20260501 08:11:28.392532 1100 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 5 messages since previous log ~2 seconds ago
I20260501 08:11:28.392561 1100 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 1 messages since previous log ~0 seconds ago
I20260501 08:11:28.392586 1100 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 1 messages since previous log ~0 seconds ago
I20260501 08:11:28.392614 1100 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 6 messages since previous log ~5 seconds ago
I20260501 08:11:28.392680 1100 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/maintenance_manager.cc:346: suppressed but not reported on 582 messages since previous log ~37 seconds ago